Metadata-Version: 2.4
Name: diffvector
Version: 0.1.0
Summary: Visual git-diff for vector embeddings — detect semantic drift before embedding model upgrades
Author-email: shunvel <shunvel@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/shunvel/diffvector
Project-URL: Repository, https://github.com/shunvel/diffvector
Project-URL: Issues, https://github.com/shunvel/diffvector/issues
Keywords: embeddings,vector-database,rag,migration,regression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: scikit-learn>=1.3
Requires-Dist: plotly>=5.18
Requires-Dist: python-dotenv>=1.0
Requires-Dist: typer>=0.9
Requires-Dist: streamlit>=1.28
Requires-Dist: chromadb>=0.4
Requires-Dist: qdrant-client>=1.7
Requires-Dist: pinecone>=5.0
Provides-Extra: models
Requires-Dist: openai>=1.0; extra == "models"
Requires-Dist: sentence-transformers>=2.2; extra == "models"
Requires-Dist: cohere>=4.0; extra == "models"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: matplotlib>=3.8; extra == "dev"
Requires-Dist: kaleido>=0.2; extra == "dev"
Provides-Extra: all
Requires-Dist: diffvector[dev,models]; extra == "all"
Dynamic: license-file

# DiffVector

[![PyPI version](https://img.shields.io/pypi/v/diffvector.svg)](https://pypi.org/project/diffvector/)
[![Docker](https://img.shields.io/badge/docker-ghcr.io-2496ED?logo=docker&logoColor=white)](https://github.com/shunvel/diffvector/pkgs/container/diffvector)
[![MIT License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/shunvel/diffvector/actions/workflows/ci.yml/badge.svg)](https://github.com/shunvel/diffvector/actions/workflows/ci.yml)

**The open-source visual git-diff for vector embeddings** — detect semantic drift and search regression before changing your embedding models or chunking strategies.

> **Author:** [shunvel](https://github.com/shunvel) · shunvel@gmail.com

---

## Install (no git clone required)

### pip — CLI + GUI

```bash
pip install diffvector

# Optional: OpenAI, Cohere, HuggingFace embedding providers
pip install "diffvector[models]"
```

```bash
# Launch the GUI
diffvector gui

# Run a CI audit
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.6 --json

# List embedding models
diffvector models
```

### Docker — GUI without Python

```bash
docker run --rm -p 8501:8501 ghcr.io/shunvel/diffvector:latest
```

Open **http://localhost:8501**.

Pass API keys and mount local Chroma data:

```bash
docker run --rm -p 8501:8501 \
  -v "$(pwd)/chroma_db:/data/chroma_db" \
  -e OPENAI_API_KEY \
  -e PINECONE_API_KEY \
  ghcr.io/shunvel/diffvector:latest
```

Qdrant on the host (Docker Desktop):

```bash
docker run --rm -p 8501:8501 \
  --add-host=host.docker.internal:host-gateway \
  -e QDRANT_URL=http://host.docker.internal:6333 \
  ghcr.io/shunvel/diffvector:latest
```

Or use the included compose stack (Qdrant + GUI):

```bash
docker compose --profile gui up --build
```

For OpenAI/Cohere/HuggingFace models inside Docker, build with model providers (~2 GB image):

```bash
docker build --build-arg INSTALL_MODELS=1 -t diffvector:full .
docker run --rm -p 8501:8501 diffvector:full
```

### From source (contributors)

```bash
git clone https://github.com/shunvel/diffvector.git
cd diffvector
pip install -e ".[dev]"
diffvector gui
```

---

## Quick Start (local testing)

After `pip install diffvector` (or editable install from source).

### Option A — Chroma only (simplest, no Docker)

```bash
# Seed local test data (zero API keys needed)
python -m utils.mock_generator

# Launch the GUI
diffvector gui
```

Open `http://localhost:8501` and use the sidebar settings below (see [Local Testing Guide](#local-testing-guide)).

### Option B — Chroma + Qdrant (Docker Desktop)

```bash
# One command: start Qdrant + seed both backends (requires git clone for scripts/)
chmod +x scripts/setup-local.sh && ./scripts/setup-local.sh

# Or with pip only:
docker compose up -d
python -m utils.mock_generator --backend all

diffvector gui
```

Or manually:

```bash
docker compose up -d
python -m utils.mock_generator --backend all
diffvector gui
```

**GUI settings for Qdrant:**

| Setting | Value |
|---------|-------|
| Vector DB | `qdrant` |
| Qdrant URL | `http://localhost:6333` |
| Collection | `demo` |
| Target model | `local/offline-hash-384` |

Qdrant dashboard: [http://localhost:6333/dashboard](http://localhost:6333/dashboard)

### Option C — Pinecone (cloud, needs API key)

```bash
cp .env.example .env
# Add: PINECONE_API_KEY=pcsk-...

python -m utils.mock_generator --backend pinecone --collection demo --count 100
diffvector gui
```

**GUI settings for Pinecone:**

| Setting | Value |
|---------|-------|
| Vector DB | `pinecone` |
| Collection / Index | `demo` |
| Pinecone API Key | from `.env` or sidebar |
| Target model | `local/offline-hash-384` |

> Pinecone has no local Docker image — the mock generator creates a serverless index named `demo` (1536-d, cosine) and upserts 100 test vectors.

> **macOS Docker note:** If `docker` is not found in a new terminal, Docker Desktop's binary may not be on your PATH:
> ```bash
> export PATH="/Applications/Docker.app/Contents/Resources/bin:$PATH"
> ```
> The included `scripts/setup-local.sh` handles this automatically (source install only).

---

## Why DiffVector?

When engineering teams upgrade embedding models (e.g., OpenAI `text-embedding-ada-002` → `text-embedding-3-small`), alter chunking strategies, or migrate across vector database vendors, they are structurally blind.

Changing models alters high-dimensional spatial distances completely. Traditional migration scripts focus on moving data pipelines, leaving developers with **zero insight** into search quality regression, semantic drift, or payload mismatch.

The result is **Silent Retrieval Decay**: your system returns mathematically valid nearest neighbors that are textually irrelevant. DiffVector catches this *before* production.

---

## How It Works

DiffVector is a **non-destructive diagnostic staging area**. It never writes to your production index.

```mermaid
flowchart LR
  subgraph sources [Your Vector DB]
    Chroma
    Qdrant
    Pinecone
  end
  subgraph engine [DiffVector Engine]
    Ingest[1. Ingest Slice]
    Simulate[2. Simulate Model]
    Diagnose[3. Diagnose Drift]
  end
  sources --> Ingest
  Ingest --> Simulate
  Simulate --> Diagnose
  Diagnose --> GUI[Streamlit GUI]
  Diagnose --> CLI[CLI / CI Gate]
```

### The 3-Step Engine

| Step | What happens |
|------|-------------|
| **1. Secure Ingestion** | Read-only pull of 200–500 document chunks + stored vectors into local memory |
| **2. Model Simulation** | Re-embed the same texts with your target model (OpenAI, Cohere, HuggingFace) |
| **3. Diagnostics** | Cluster shift map (t-SNE) + Top-K retrieval overlap matrix |

---

## CLI (CI Gate)

Gate embedding upgrades in your pipeline:

```bash
# Audit with JSON output — exits 1 if overlap below threshold
diffvector audit \
  --source chroma \
  --path ./chroma_db \
  --collection demo \
  --target-model local/offline-hash-384 \
  --n 300 \
  --topk 5 \
  --min-overlap 0.6 \
  --json

# List supported models
diffvector models
```

### Canary queries (optional)

Provide a source model to embed free-text queries in both spaces:

```bash
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model text-embedding-3-small \
  --source-model text-embedding-ada-002 \
  --queries canaries.txt \
  --min-overlap 0.5
```

---

## Local Testing Guide

This section covers zero-config local testing — no production vector DB, no OpenAI/Cohere keys, and no HuggingFace download required.

### Which database can I use locally?

You do **not** need an existing production database. DiffVector ships a mock data generator:

| Backend | How to set up | Docker needed? |
|---------|---------------|----------------|
| **Chroma** (recommended) | `python -m utils.mock_generator` → creates `./chroma_db` | **No** — embedded SQLite via Python |
| **Qdrant** | `docker compose up -d` then `python -m utils.mock_generator --backend qdrant` | **Yes** (Docker Desktop) |
| **Pinecone** | `python -m utils.mock_generator --backend pinecone` (needs `PINECONE_API_KEY`) | No (cloud API) |

**Start with Chroma** for the fastest path. Use Qdrant when you want to validate the Qdrant connector specifically. Use Pinecone when you need to verify the cloud connector before production migration.

### Which test command should I run?

| Command | What it runs | API keys | Docker |
|---------|--------------|----------|--------|
| `pytest tests/ -v` | Unit tests (connectors, embeddings, analytics) | None | No |
| `./scripts/test-all-backends.sh` | Seed + CLI audit for **each** backend | Pinecone only | Qdrant |
| `./scripts/test-cli.sh` | Unit tests + CI gate checks + optional Qdrant/Pinecone | Pinecone only | Qdrant (optional) |
| `diffvector gui` | Manual GUI smoke test | Per backend | Qdrant only |

Run **`pytest`** during development. Run **`test-all-backends.sh`** before a release when you want all three connectors verified in one shot. Run **`test-cli.sh`** for the full pre-publish gate (exit codes, empty collection, JSON schema).

### GUI walkthrough (3 steps)

Open **http://localhost:8501** after `diffvector gui`.

#### Step 1 — Ingest (Chroma)

| Sidebar setting | Value |
|-----------------|-------|
| Vector DB | `chroma` |
| Chroma path | `./chroma_db` |
| Collection | `demo` |
| Sample size | `300` (default) |

Click **Connect & Ingest** on tab **① Ingest**.

#### Step 1 — Ingest (Qdrant)

| Sidebar setting | Value |
|-----------------|-------|
| Vector DB | `qdrant` |
| Qdrant URL | `http://localhost:6333` |
| Collection | `demo` |

Click **Connect & Ingest**.

#### Step 1 — Ingest (Pinecone)

Seed first (creates index `demo` if missing):

```bash
python -m utils.mock_generator --backend pinecone --collection demo --count 100
```

| Sidebar setting | Value |
|-----------------|-------|
| Vector DB | `pinecone` |
| Collection / Index | `demo` |
| Pinecone API Key | `PINECONE_API_KEY` from `.env` or sidebar |
| Namespace | leave blank (default) |

Click **Connect & Ingest**. Verify in the [Pinecone console](https://app.pinecone.io/) that index **`demo`** has vectors.

#### Step 2 — Simulate

| Sidebar setting | Value |
|-----------------|-------|
| Target model | `local/offline-hash-384` |
| OpenAI / Cohere / HF keys | **leave blank** |

Click **Run Simulation** on tab **② Simulate**.

#### Step 3 — Diagnose

Click **Run Diagnostics** on tab **③ Diagnose** to see the cluster shift map and retrieval overlap matrix.

### Why are API keys in the sidebar?

The sidebar lists **all supported integrations** in one place. Keys are **optional** for local testing:

| Key | When you actually need it |
|-----|---------------------------|
| **OpenAI** | Target model is `text-embedding-ada-002`, `text-embedding-3-small`, etc. — or canary queries with an OpenAI source model |
| **Cohere** | Target model is `embed-english-v3.0` |
| **HuggingFace (`HF_TOKEN`)** | Target model is `sentence-transformers/all-MiniLM-L6-v2` or `all-mpnet-base-v2` and HuggingFace blocks the download (403) |
| **Qdrant / Pinecone** | Only when connecting to those cloud or secured instances |

For first-time local testing: **Chroma + `local/offline-hash-384` + no keys**.

### Troubleshooting: `403 Forbidden` on Simulate (Step 2)

This error means HuggingFace blocked the download of a sentence-transformers model (rate limits, no token, or network/proxy restrictions).

**Fix (pick one):**

1. **Use the offline model (recommended for local testing)**
   - Target model → `local/offline-hash-384`
   - No download, no API keys, works immediately

2. **Use a cached HuggingFace model**
   - If you previously downloaded MiniLM on this machine, select `sentence-transformers/all-MiniLM-L6-v2` — DiffVector loads from cache first

3. **Add a HuggingFace token**
   ```bash
   # In .env or the sidebar "HuggingFace Token" field
   HF_TOKEN=hf_your_token_here
   ```
   Get a free token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens), or run:
   ```bash
   huggingface-cli login
   ```

### Expected results with mock data

Low overlap (e.g. ~10%) is **normal** with the mock generator. It stores **random** 1536-d vectors as "old" embeddings, while your target model produces **semantic** vectors from the text. DiffVector is correctly detecting drift — not a bug.

Real production audits (same texts, ada-002 stored vectors vs text-embedding-3-small re-embed) will show more meaningful overlap numbers.

### CLI local testing

```bash
# Chroma — offline model, no keys
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Qdrant — requires Docker
diffvector audit \
  --source qdrant --url http://localhost:6333 --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Pinecone — requires PINECONE_API_KEY in .env
python -m utils.mock_generator --backend pinecone --collection demo --count 100
diffvector audit \
  --source pinecone --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json
```

### Test all backends at once (before publish)

```bash
pip install -e ".[dev]"

# One script: seed + audit Chroma, Qdrant, and Pinecone
chmod +x scripts/test-all-backends.sh
./scripts/test-all-backends.sh
```

What it does:

1. Seeds `./chroma_db` and runs `diffvector audit --source chroma`
2. Starts Qdrant via Docker, seeds collection `demo`, runs audit
3. Seeds Pinecone index `demo` and runs audit — **skipped with a warning** if `PINECONE_API_KEY` is not in `.env`

Override defaults:

```bash
COLLECTION=my-index TARGET_MODEL=local/offline-hash-384 ./scripts/test-all-backends.sh
```

Full pre-publish gate (unit tests + CI exit-code checks + optional backends):

```bash
chmod +x scripts/test-cli.sh
./scripts/test-cli.sh
```

### Run unit tests

```bash
pip install -e ".[dev]"
pytest tests/ -v
```

All unit tests run without API keys or Docker (connectors are tested against local Chroma fixtures and mocked Pinecone factory paths).

### Seed all backends

```bash
# Chroma + Qdrant always; Pinecone only if PINECONE_API_KEY is set
python -m utils.mock_generator --backend all --collection demo --count 100
```

### Restarting your local stack

```bash
# Full reset: Qdrant + seed both Chroma and Qdrant
./scripts/setup-local.sh

# Or step by step
export PATH="/Applications/Docker.app/Contents/Resources/bin:$PATH"  # macOS if needed
docker compose up -d
python -m utils.mock_generator --backend all
diffvector gui
```

---

## Supported Backends

| Backend | Connection |
|---------|-----------|
| **ChromaDB** | Local persistent path or in-memory |
| **Qdrant** | `QDRANT_URL` + optional `QDRANT_API_KEY` |
| **Pinecone** | `PINECONE_API_KEY` + index name + namespace |

## Supported Target Models

| Provider | Models |
|----------|--------|
| **Local (offline)** | `local/offline-hash-384` — no download, no API keys (recommended for first run) |
| OpenAI | `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large` |
| Cohere | `embed-english-v3.0` |
| HuggingFace | `all-MiniLM-L6-v2` (384-d), `all-mpnet-base-v2` (768-d) |

Copy `.env.example` to `.env` and fill in keys only when needed:

```bash
cp .env.example .env
```

| Variable | Purpose |
|----------|---------|
| `OPENAI_API_KEY` | OpenAI embedding models |
| `COHERE_API_KEY` | Cohere embedding models |
| `HF_TOKEN` | HuggingFace model downloads (fixes 403 on sentence-transformers) |
| `QDRANT_URL` / `QDRANT_API_KEY` | Qdrant Cloud or secured instances |
| `PINECONE_API_KEY` | Pinecone indexes |

---

## Key Metrics

- **Top-K Overlap** — `|old_top_k ∩ new_top_k| / k`, bounded [0.0, 1.0]
- **Drift Score** — `1 − mean_overlap`
- **High-Risk Flag** — overlap < 40% per query

Cross-dimension comparison (1536-d → 384-d) works because retrieval happens **inside each space independently**, then results are compared by document ID.

---

## Screenshots

<p align="center">
  <img src="docs/screenshots/streamlit-workflow.png" alt="DiffVector 3-step Streamlit workflow" width="900"/>
</p>

<p align="center">
  <em>The 3-step engine: ingest a read-only slice, simulate a target embedding model, diagnose drift.</em>
</p>

<p align="center">
  <img src="docs/screenshots/diagnostic-metrics.png" alt="Diagnostic metrics panel" width="900"/>
  &nbsp;&nbsp;
  <img src="docs/screenshots/cluster-shift-map.png" alt="Cluster shift map" width="900"/>
</p>

<p align="center">
  <em>Step 3 metrics and cluster shift map (mock Chroma data → <code>local/offline-hash-384</code>).</em>
</p>

<p align="center">
  <img src="docs/screenshots/cli-audit.png" alt="CLI audit JSON output" width="900"/>
</p>

<p align="center">
  <em>CLI audit with <code>--json</code> for CI pipelines.</em>
</p>

Regenerate screenshots after UI changes:

```bash
pip install -e ".[dev]"
python scripts/generate_readme_assets.py
```

---

## Pre-publish checklist

Run this **before making the repo public**:

```bash
pip install -e ".[dev]"

# 1. All backends — seed + audit in one run (Pinecone needs PINECONE_API_KEY in .env)
chmod +x scripts/test-all-backends.sh
./scripts/test-all-backends.sh

# 2. Full CI gate — unit tests, exit codes, empty collection, JSON schema
chmod +x scripts/test-cli.sh
./scripts/test-cli.sh

# 3. Refresh README screenshots
python scripts/generate_readme_assets.py

# 4. Manual GUI smoke test (optional)
python -m utils.mock_generator --backend all   # Chroma + Qdrant (+ Pinecone if key set)
diffvector gui
```

### What `test-cli.sh` verifies

| Check | What it proves |
|-------|----------------|
| `pytest tests/ -q` | Core engine tests (connectors, embeddings, analytics) |
| Chroma seed | Mock data exists at `./chroma_db` |
| `diffvector models` | CLI entry point works |
| `diffvector audit --json` | End-to-end Chroma ingest → embed → overlap |
| Exit code `1` on `--min-overlap 0.99` | CI gate fails when drift is too high |
| Empty collection | Graceful error, not a crash |
| Qdrant audit (if Docker up) | Qdrant seed + connector + CLI path |
| Pinecone audit (if `PINECONE_API_KEY` set) | Pinecone seed + connector + CLI path |

### What `test-all-backends.sh` verifies

Same seed + audit path as above, but **requires** Chroma and Qdrant to pass (exits on failure). Pinecone is tested when a key is present; otherwise it prints a skip warning and still exits 0 after Chroma + Qdrant pass.

### Qdrant seed data (already included)

Qdrant test data is seeded by the mock generator — no manual upserts needed:

```bash
# Start Qdrant + seed Chroma and Qdrant together
./scripts/setup-local.sh

# Or Qdrant only
docker compose up -d
python -m utils.mock_generator --backend qdrant --url http://localhost:6333 --collection demo --count 300
```

Verify in the Qdrant dashboard: [http://localhost:6333/dashboard](http://localhost:6333/dashboard) → collection **`demo`** (~300 points, 1536-d cosine).

### Manual CLI commands to spot-check

```bash
# Must pass (exit 0)
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Must fail (exit 1) — proves CI gate works
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.99 --json

# Qdrant (after docker compose up -d + seed)
diffvector audit \
  --source qdrant --url http://localhost:6333 --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Pinecone (after seed + PINECONE_API_KEY in .env)
diffvector audit \
  --source pinecone --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json
```

---

## Project Structure

```
diffvector/
├── app.py                   # Streamlit GUI
├── cli.py                   # Typer CLI (diffvector audit)
├── core/
│   ├── connectors.py        # Chroma, Qdrant, Pinecone (read-only)
│   ├── embeddings.py        # OpenAI, Cohere, sentence-transformers
│   └── analytics.py         # t-SNE, Top-K overlap
├── utils/
│   └── mock_generator.py    # Seed Chroma / Qdrant / Pinecone for testing
├── docker-compose.yml       # Local Qdrant + optional GUI profile
├── Dockerfile               # ghcr.io/shunvel/diffvector image
├── scripts/
│   ├── setup-local.sh              # Start Qdrant + seed Chroma & Qdrant
│   ├── test-cli.sh                 # Pre-publish CLI verification
│   ├── test-all-backends.sh        # Chroma + Qdrant + Pinecone in one run
│   └── generate_readme_assets.py   # Regenerate docs/screenshots/
├── docs/
│   └── screenshots/                # README images (auto-generated)
└── tests/                   # TV-001 through TV-004
```

---

## Security

- **Read-only** — DiffVector never modifies your vector database
- **Local-first** — Data stays in Streamlit session state or CLI stdout
- **Keys stay local** — API keys via `.env` or sidebar; never logged

---

## Development

```bash
git clone https://github.com/shunvel/diffvector.git
cd diffvector
pip install -e ".[dev]"
pytest tests/ -v
diffvector gui
```

Publishing: see [docs/PUBLISH.md](docs/PUBLISH.md).

---

## License

MIT © 2026 [shunvel](https://github.com/shunvel)
