Metadata-Version: 2.4
Name: zeusdb-vector-database
Version: 0.9.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Dist: numpy>=2.2.6,<3.0.0
Requires-Dist: maturin>=1.9.3 ; extra == 'dev'
Requires-Dist: pytest>=8.4.0 ; extra == 'dev'
Requires-Dist: pytest-xdist>=3.8.0 ; extra == 'dev'
Requires-Dist: psutil>=7.0.0 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
License-File: NOTICE
Summary: Blazing-fast vector DB with real-time similarity search and metadata filtering.
Author-email: ZeusDB <contact@zeusdb.com>
License-Expression: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://github.com/zeusdb/zeusdb-vector-database/issues
Project-URL: Documentation, https://docs.zeusdb.com/en/latest/vector_database/
Project-URL: Repository, https://github.com/zeusdb/zeusdb-vector-database

<p align="center" width="100%">
  <img src="https://github.com/user-attachments/assets/ad21baec-6f4c-445c-b423-88a081ca2b97" alt="zeusdb-vector-database-logo-cropped" />
  <h1 align="center">ZeusDB Vector Database</h1>
</p>

<!-- <h2 align="center">Fast, Rust-powered vector database for similarity search</h2> -->
<!--**Fast, Rust-powered vector database for similarity search** -->

<!-- badges: start -->

<div align="center">
  <table>
    <tr>
      <td><strong>Meta</strong></td>
      <td>
        <a href="https://pypi.org/project/zeusdb-vector-database/"><img src="https://img.shields.io/pypi/v/zeusdb-vector-database?label=PyPI&color=blue"></a>&nbsp;
        <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%7C3.11%7C3.12%7C3.13%7C3.14-blue?logo=python&logoColor=ffdd54"></a>&nbsp;
        <a href="https://github.com/zeusdb/zeusdb-vector-database/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"></a>&nbsp;
        <a href="https://www.rust-lang.org"><img src="https://img.shields.io/badge/Powered%20by-Rust-black?logo=rust&logoColor=white" alt="Powered by Rust"></a>&nbsp;
        <a href="https://github.com/ZeusDB"><img src="https://github.com/user-attachments/assets/e140d900-1160-4eaa-85c0-2b3507a5f0f5" alt="ZeusDB"></a>&nbsp;
      </td>
    </tr>
  </table>
</div>

<!-- badges: end -->

<br />

## ℹ️ What is ZeusDB Vector Database?

ZeusDB Vector Database is a high-performance, Rust-powered vector database designed for fast similarity search across high-dimensional data. It enables efficient approximate nearest neighbor (ANN) search, ideal for use cases like document retrieval, semantic search, recommendation systems, and AI-powered assistants.

ZeusDB leverages the HNSW (Hierarchical Navigable Small World) algorithm for speed and accuracy, with native Python bindings for easy integration into data science and machine learning workflows. Whether you're indexing millions of vectors or running low-latency queries in production, ZeusDB offers a lightweight, extensible foundation for scalable vector search.

<br/>

## ⭐ Features

🐍 User-friendly Python API for adding vectors and running similarity searches

🔥 High-performance Rust backend optimized for speed and concurrency

🔍 Approximate Nearest Neighbor (ANN) search using HNSW for fast, accurate results

📦 Product Quantization (PQ) for compact storage and faster distance computations

📥 Flexible input formats, including native Python types and NumPy arrays

🗂️ Metadata-aware filtering for precise and contextual querying

💾 Save and load complete indexes to disk

<br/>

## ✅ Supported Distance Metrics

ZeusDB Vector Database supports the following metrics for vector similarity search. All metric names are case-insensitive, so "cosine", "COSINE", and "Cosine" are treated identically.

| Metric | Description                          | Accepted Values (case-insensitive)  | Quantization |
|--------|--------------------------------------|--------|--------|
| cosine | Cosine Distance (1 - Cosine Similarity) | "cosine", "COSINE", "Cosine" | supported |
| l1     | Manhattan distance                   | "l1", "L1" | refused |
| l2     | Euclidean distance                 | "l2", "L2" | supported |
| dot    | Inner product, reported as 1 - dot | "dot", "DOT" | refused |

`create()` raises on a refused pair rather than building an index that ranks by the wrong quantity, and `load()` refuses a saved directory that pairs them.

### 📏 Scores vs Distances

All distance metrics in ZeusDB Vector Database return distance values, not similarity scores:

 - Lower values = more similar
 - A vector identical to the query scores 0.0, or a value within floating point error of it

This applies to all distance types, including cosine. `dot` is the one exception to the zero, because its score is `1 - dot` and an inner product above one takes it below zero.

Under `cosine`, vectors are normalized to unit length when they are stored. A vector you read back with `return_vector=True` or `get_records()` is therefore the normalized form, not the values you supplied. Under `l1`, `l2` and `dot` the values are stored unchanged.

A zero vector has no direction, so under `cosine` it sits at distance 1.0 from everything, including itself.

**On a quantized index the score is a distance to the record's reconstruction, not to the vector you inserted.** Under `l2` it is the euclidean distance to that reconstruction and under `cosine` it is the cosine distance to it, so either way the number is on the scale a raw index of the same space reports and the two are comparable. It is not equal to the raw score, because the index no longer holds the vector you gave it, and the difference is the quantization error. Rerank replaces it with an exact distance to the raw vector, and it is on by default for `quantized_with_raw`.

```python
from zeusdb_vector_database import VectorDatabase

index = VectorDatabase().create("hnsw", dim=4, space="cosine")
index.add({"id": "a", "values": [1.0, 0.0, 0.0, 0.0]})
print(round(index.search([1.0, 0.0, 0.0, 0.0], top_k=1)[0]["score"], 6))
```

*Output*
```
0.0
```

<br/>

## 📦 Installation

You can install ZeusDB Vector Database with 'uv' or alternatively using 'pip'.

### Recommended (with uv):
```bash
uv pip install zeusdb-vector-database
```

### Alternatively (using pip):
```bash
pip install zeusdb-vector-database
```

<br/>

## 🔥 Quick Start Example

```python
# Import the vector database module
from zeusdb_vector_database import VectorDatabase

# Instantiate the VectorDatabase class
vdb = VectorDatabase()

# Initialize and set up the database resources
index = vdb.create(index_type="hnsw", dim=8)

# Vector embeddings with accompanying ID's and Metadata
records = [
    {"id": "doc_001", "values": [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7], "metadata": {"author": "Alice"}},
    {"id": "doc_002", "values": [0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9], "metadata": {"author": "Bob"}},
    {"id": "doc_003", "values": [0.11, 0.21, 0.31, 0.15, 0.41, 0.22, 0.61, 0.72], "metadata": {"author": "Alice"}},
    {"id": "doc_004", "values": [0.85, 0.15, 0.42, 0.27, 0.83, 0.52, 0.33, 0.95], "metadata": {"author": "Bob"}},
    {"id": "doc_005", "values": [0.12, 0.22, 0.33, 0.13, 0.45, 0.23, 0.65, 0.71], "metadata": {"author": "Alice"}},
]

# Upload records using the `add()` method
add_result = index.add(records)
print(add_result.summary())

# Perform a similarity search and print the top 2 results
query_vector = [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7]

results = index.search(vector=query_vector, filter=None, top_k=2)

for i, res in enumerate(results, 1):
    print(f"{i}. ID: {res['id']}, Score: {res['score']:.6f}, Metadata: {res['metadata']}")
```

*Results Output:*
```
5 inserted, 0 errors
1. ID: doc_001, Score: 0.000000, Metadata: {'author': 'Alice'}
2. ID: doc_003, Score: 0.000988, Metadata: {'author': 'Alice'}
```

`add_result.summary()` returns a plain ASCII string, so it prints on any console encoding. The same counts are on `add_result.total_inserted` and `add_result.total_errors` if you want the numbers rather than the sentence.

<br/>

## ✨ Usage

ZeusDB Vector Database makes it easy to work with high-dimensional vector data using a fast, memory-efficient HNSW index. Whether you're building semantic search, recommendation engines, or embedding-based clustering, the workflow is simple and intuitive.

**Three simple steps**

1. **Create an index** using `.create()`
2. **Add data** using `.add(...)`
3. **Conduct a similarity search** using `.search(...)`

Each step is covered below.

<br/>

### 1️⃣ Create an Index

To get started, first initialize a VectorDatabase and create an HNSWIndex. You can configure the vector dimension, distance metric, and graph construction parameters.

```python
# Import the vector database module
from zeusdb_vector_database import VectorDatabase

# Instantiate the VectorDatabase class
vdb = VectorDatabase()

# Initialize and set up the database resources
index = vdb.create(
    index_type="hnsw",
    dim=8,
    space="cosine",
    m=16,
    ef_construction=200,
    expected_size=5,
)
print(index.info())
```

*Output*
```
HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=5, vectors=0, quantization=none)
```

<br/>

#### 📘 Parameters - `create()`

| Parameter        | Type   | Default   | Description                                                                 |
|------------------|--------|-----------|-----------------------------------------------------------------------------|
| `index_type`     | `str`  | `"hnsw"`  | The type of vector index to create. Currently only `"hnsw"` is supported. Case-insensitive. |
| `dim`            | `int`  | *required* | Dimensionality of the vectors to be indexed, from 1 to 65,536. Each vector must have this length. **Required as of 0.8.0**, see below. |
| `space`          | `str`  | `"cosine"`| Distance metric used for similarity search. One of `"cosine"`, `"l1"`, `"l2"`, `"dot"`. Case-insensitive. `"l1"` and `"dot"` cannot be combined with `quantization_config`. |
| `m`              | `int`  | `16` or `32`, see below | Number of bi-directional connections created for each new node, from 2 to 256. Higher `m` improves recall but increases index size and build time. |
| `ef_construction`| `int`  | `200`     | Width of the candidate search each insertion runs, from 1 to 4,096. It costs build time and buys graph quality, and it changes neither search latency nor the size of the finished index. See below. |
| `expected_size`  | `int`  | `10000`   | Estimated number of records to be inserted, from 1 to 100,000,000. Used for preallocating internal data structures and for choosing the default `m`. Not a hard limit, see below. |
| `quantization_config` | `dict` | `None` | Product Quantization configuration for memory-efficient vector compression. See [Product Quantization](#️-product-quantization). |
| `indexed_fields` | `list[str]` | `None` | Metadata fields to build a column for, so that a filter naming only those fields does not read every record. Up to 32 names, no duplicates, none of `$and`, `$or` or `$not`. See [Declaring the fields you filter on](#-declaring-the-fields-you-filter-on). |

**`dim` is required.** There is no default, because `dim` has to equal the width your embedding model produces and an index built at any other width rejects every vector you add to it. Omitting it raises `TypeError`.

```python
try:
    vdb.create("hnsw")
except TypeError as error:
    print(error)
```

*Output*
```
create() requires 'dim', the width of the vectors this index will hold. There is no default because dim has to equal the width your embedding model produces, and an index built at any other width rejects every vector you add to it. Read it off one embedding with len(vector), or pass the width your model documents, for example dim=1536 for OpenAI text-embedding-3-small or dim=768 for most sentence-transformers models.
```

**The default `m` depends on `expected_size`.** It is 16 for an `expected_size` of 25,000 or less, and 32 above that. A graph too sparse for the number of records loses recall that no search width recovers, so declare `expected_size` honestly or set `m` yourself. Passing `m` explicitly always wins, and [`rebuild()`](#-change-m-after-the-fact-with-rebuild) changes it afterwards.

```python
vdb.create("hnsw", dim=8, expected_size=25_000).get_stats()["m"]   # '16'
vdb.create("hnsw", dim=8, expected_size=25_001).get_stats()["m"]   # '32'
```

**`expected_size` is a hint and not a limit.** An index accepts more records than it declared and the graph grows to fit them. What it does not change is `m`, which [`rebuild()`](#-change-m-after-the-fact-with-rebuild) does. Passing twice the declared size logs a warning once, on the `add()` that crosses it.

**`ef_construction` costs build time and buys graph quality.** It changes neither search latency nor the size of the finished index. Build time is linear in it above 100, so 50,000 records of `dim=1536` build in 76.9 s at the default and 262.1 s at 800. Recall stops improving at or near the default on most data, and where it keeps climbing a larger `ef_search` buys more for less, so raise `ef_search` before raising this. The default does not move with `m`, so 200 is 6.25 times the layer zero neighbour budget of 32 at `m=16` and 3.125 times the budget of 64 at `m=32`. Measured at 50,000 records with `m=32`, raising it to 400 bought 0.0010 recall at 10 on OpenAI embeddings of `dim=1536`, 0.0002 on SIFT and 0.0078 on GloVe, for 1.9 to 2.2 times the build time, which is why the constant stays.

**Keep `ef_construction` above `2 × m`.** At or below the neighbour budget the graph keeps every candidate the insertion search returned and prunes none of them. `create()` and [`rebuild()`](#-change-m-after-the-fact-with-rebuild) both warn when the pair reaches that point, and both take the pair, so either remedy the warning names can be taken. The defaults are clear of it.

<br/>

### 2️⃣ Add Data to the Index

ZeusDB provides a flexible `.add(...)` method that supports multiple input formats for inserting or updating vectors in the index. Whether you're adding a single record, a list of documents, or structured arrays, the API is designed to be both intuitive and robust. Each record can include optional metadata for filtering or downstream use.

All formats return an `AddResult` containing `total_inserted`, `total_errors`, `errors`, `vector_shape` and `ids`.

#### ✅ Format 1 – Single Object

```python
index = vdb.create("hnsw", dim=2)

add_result = index.add({
    "id": "doc1",
    "values": [0.1, 0.2],
    "metadata": {"text": "hello"}
})

print(add_result.total_inserted, add_result.total_errors)
print(add_result.is_success())
```

*Output*
```
1 0
True
```

#### ✅ Format 2 – List of Objects

```python
index = vdb.create("hnsw", dim=2)

add_result = index.add([
    {"id": "doc1", "values": [0.1, 0.2], "metadata": {"text": "hello"}},
    {"id": "doc2", "values": [0.3, 0.4], "metadata": {"text": "world"}},
])

print(add_result.total_inserted, add_result.total_errors)
print(add_result.vector_shape)
print(add_result.errors)
```

*Output*
```
2 0
(2, 2)
[]
```

#### ✅ Format 3 – Separate Arrays

```python
index = vdb.create("hnsw", dim=2)

add_result = index.add({
    "ids": ["doc1", "doc2"],
    "embeddings": [[0.1, 0.2], [0.3, 0.4]],
    "metadatas": [{"text": "hello"}, {"text": "world"}],
})
print(add_result)
```

*Output*
```
AddResult(inserted=2, errors=0, shape=Some((2, 2)))
```

The `Some(...)` wrapper appears only in the printed form. `add_result.vector_shape` is the plain tuple `(2, 2)`.

#### ✅ Format 4 – Using NumPy Arrays

ZeusDB also supports NumPy arrays as input for seamless integration with scientific and ML workflows.

```python
import numpy as np

index = vdb.create("hnsw", dim=4)

data = [
    {"id": "doc2", "values": np.array([0.1, 0.2, 0.3, 0.4], dtype=np.float32), "metadata": {"type": "blog"}},
    {"id": "doc3", "values": np.array([0.5, 0.6, 0.7, 0.8], dtype=np.float32), "metadata": {"type": "news"}},
]

result = index.add(data)

print(result.total_inserted, result.total_errors)
```

*Output*
```
2 0
```

#### ✅ Format 5 – Separate Arrays with NumPy

```python
index = vdb.create("hnsw", dim=2)

add_result = index.add({
    "ids": ["doc1", "doc2"],
    "embeddings": np.array([[0.1, 0.2], [0.3, 0.4]], dtype=np.float32),
    "metadatas": [{"text": "hello"}, {"text": "world"}],
})
print(add_result)
```

*Output*
```
AddResult(inserted=2, errors=0, shape=Some((2, 2)))
```

Each format is parsed and validated automatically. Invalid records are skipped rather than aborting the call, and the reason for each is returned in `errors`. A record whose vector contains `NaN` or an infinity is rejected this way.

<br/>

#### ⚠️ Adding an ID that already exists

`add()` upserts by default. Re-adding an existing ID **replaces the whole record**, metadata included. Metadata is not merged, so a key you leave out of the new record is gone.

```python
index = vdb.create("hnsw", dim=2)
index.add({"id": "doc1", "values": [0.1, 0.2], "metadata": {"text": "hello", "lang": "en"}})

# "lang" is not carried over
index.add({"id": "doc1", "values": [0.3, 0.4], "metadata": {"text": "goodbye"}})
print(index.get_records("doc1", return_vector=False))

# overwrite=False rejects the record instead, and counts it as an error
rejected = index.add({"id": "doc1", "values": [0.5, 0.6]}, overwrite=False)
print(rejected.total_inserted, rejected.total_errors)
print(rejected.errors)
```

*Output*
```
[{'id': 'doc1', 'metadata': {'text': 'goodbye'}}]
0 1
["Vector doc1: ValueError: Vector with ID 'doc1' already exists"]
```

A rejected record is reported in the `AddResult`. It does not raise. The rejection is also logged at WARNING level, which is visible on stderr under the default development settings.

Every overwrite leaves a node behind in the graph. See [`compact()`](#️-reclaim-space-left-by-removals-and-overwrites).

<br/>

#### 📘 Parameters - `add()`

The `add()` method inserts or replaces one or more vectors in the index.

| Parameter | Type                                | Default | Description |
|-----------|-------------------------------------|---------|-------------|
| `data`    | `dict`, `list[dict]`, `dict` of arrays, or `np.ndarray` | *required* | Input records to upsert into the index. Supports the five formats above. |
| `overwrite` | `bool`                            | `True`  | Whether an ID already in the index is replaced. With `False`, a colliding record is skipped and counted as an error. |

**The parallel arrays must be the same length.** Formats 3 and 5 pair `ids[i]` with `vectors[i]` and `metadatas[i]` by position, so a disagreement in length is a caller error and raises `ValueError` naming both lengths and which field is short. Nothing is inserted before the raise, so the call is safe to retry.

```python
two_wide = vdb.create("hnsw", dim=2)
try:
    two_wide.add({"ids": ["c", "d", "e"], "embeddings": [[0.1, 0.2], [0.3, 0.4]]})
except ValueError as error:
    print(error)
print(len(two_wide))
```

*Output*
```
add received 3 entries under 'ids' and 2 under 'embeddings'. A batch pairs them by position, so the two must be the same length, and 'embeddings' is the short one. Supply one id per vector, or omit 'ids' entirely.
0
```

The rule covers `ids`, `metadatas` and `metadata`, under every spelling of the vector key, on both the list and the NumPy branch. Omitting `ids` entirely is not a disagreement and still generates one per record. A parallel array must be a `list`; a `tuple` or an `ndarray` raises `TypeError`.

**A batch is not atomic, and which failures raise is deliberate.** A malformed *batch* raises before anything is inserted, so the call is safe to retry: that covers parallel arrays of different lengths, a parallel array of the wrong type, and an input that is not one of the five formats. A malformed *record* inside a well formed batch does not raise. It is counted in `total_errors`, described in `errors`, and the records around it are inserted: that covers a vector of the wrong width, a non-finite value, and a collision under `overwrite=False`. Check `is_success()` rather than assuming the call either inserted everything or nothing.

**Returns:**
`AddResult` with:
- `total_inserted`: number of records successfully inserted or replaced
- `total_errors`: number of failed records
- `errors`: list of error messages
- `vector_shape`: the shape of the processed batch, as `(rows, dim)`
- `ids`: the ID of every record that was inserted or replaced, in insertion order
- `is_success()`: `True` when `total_errors` is zero
- `summary()`: a one-line string of the two counts

`ids` is how you learn the IDs the index generated for records you supplied without one. It lines up with `total_inserted` and with nothing else, so a rejected record contributes no ID and `errors` is what names it.

```python
index = vdb.create("hnsw", dim=2)
generated = index.add({"vectors": [[0.1, 0.2], [0.3, 0.4]]})
print(generated.ids)

supplied = index.add({"ids": ["a", "b"], "embeddings": [[0.5, 0.6], [0.7, 0.8]]})
print(supplied.ids)

partial = index.add({"ids": ["ok", "bad"], "embeddings": [[0.1, 0.2], [0.1]]})
print(partial.ids, partial.total_inserted, partial.total_errors)
```

*Output*
```
['vec_1', 'vec_2']
['a', 'b']
['ok'] 1 1
```

<br/>

### 3️⃣ Conduct a Similarity Search

Query the index using a new vector and retrieve the top-k nearest neighbors. You can also filter by metadata or return the stored vectors.

The examples below all run against this index:

```python
index = vdb.create(index_type="hnsw", dim=8)
index.add([
    {"id": "doc_001", "values": [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7], "metadata": {"author": "Alice"}},
    {"id": "doc_002", "values": [0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9], "metadata": {"author": "Bob"}},
    {"id": "doc_003", "values": [0.11, 0.21, 0.31, 0.15, 0.41, 0.22, 0.61, 0.72], "metadata": {"author": "Alice"}},
    {"id": "doc_004", "values": [0.85, 0.15, 0.42, 0.27, 0.83, 0.52, 0.33, 0.95], "metadata": {"author": "Bob"}},
    {"id": "doc_005", "values": [0.12, 0.22, 0.33, 0.13, 0.45, 0.23, 0.65, 0.71], "metadata": {"author": "Alice"}},
])
query_vector = [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7]
```

#### 🔍 Search Example 1 - Basic (Returning Top 2 most similar)

```python
results = index.search(vector=query_vector, top_k=2)
for res in results:
    print(res["id"], round(res["score"], 6), res["metadata"])
```

*Output*
```
doc_001 0.0 {'author': 'Alice'}
doc_003 0.000988 {'author': 'Alice'}
```

#### 🔍 Search Example 2 - Query with metadata filter

```python
results = index.search(vector=query_vector, filter={"author": "Alice"}, top_k=5)
for res in results:
    print(res["id"], round(res["score"], 6), res["metadata"])
```

*Output*
```
doc_001 0.0 {'author': 'Alice'}
doc_003 0.000988 {'author': 'Alice'}
doc_005 0.001143 {'author': 'Alice'}
```

**The filter decides which records are ranked, not which results survive.** A search asking for five results with a filter matching a hundred records returns the five nearest of those hundred. `top_k` is the page size and nothing else, so there is no need to raise it when you filter. See [Metadata Filtering](#️-metadata-filtering) for what that costs.

#### 🔍 Search Example 3 - Search results include vectors

Set `return_vector=True` to get the stored embedding alongside the metadata and score. Under `cosine` this is the normalized vector, not the values you supplied.

The vector is a `list` of Python floats, from both `search` and `get_records`.

```python
results = index.search(vector=query_vector, top_k=1, return_vector=True)
print(results[0]["id"], round(results[0]["score"], 6))
print([round(v, 4) for v in results[0]["vector"]])
```

*Output*
```
doc_001 0.0
[0.0913, 0.1826, 0.2739, 0.0913, 0.3651, 0.1826, 0.5477, 0.639]
```

#### 🔍 Search Example 4 - Batch Search with a list of vectors

Perform a similarity search on multiple query vectors at once. The result is a list of result lists, one per query, in the order the queries were given.

```python
batch = [
    [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7],
    [0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9],
]
results = index.search(vector=batch, top_k=2)
for q, hits in enumerate(results):
    print(f"query {q}:", [(h["id"], round(h["score"], 6)) for h in hits])
```

*Output*
```
query 0: [('doc_001', 0.0), ('doc_003', 0.000988)]
query 1: [('doc_002', 0.0), ('doc_004', 0.002238)]
```

#### 🔍 Search Example 5 - Batch Search with NumPy Array

```python
query_batch = np.array(batch, dtype=np.float32)

results = index.search(vector=query_batch, top_k=2)
for q, hits in enumerate(results):
    print(f"query {q}:", [h["id"] for h in hits])
```

*Output*
```
query 0: ['doc_001', 'doc_003']
query 1: ['doc_002', 'doc_004']
```

#### 🔍 Search Example 6 - Batch Search with metadata filter

The same filter is applied to every query in the batch. Each query gets the two nearest of Alice's documents, which for the second query are not among its two nearest documents overall.

```python
results = index.search(batch, filter={"author": "Alice"}, top_k=2)
for q, hits in enumerate(results):
    print(f"query {q}:", [h["id"] for h in hits])
```

*Output*
```
query 0: ['doc_001', 'doc_003']
query 1: ['doc_005', 'doc_003']
```

<br/>

#### 📘 Parameters - `search()`

The `search()` method retrieves the top-k most similar vectors from the index given an input query vector. Results include the vector ID, distance score, metadata, and optionally the stored vector.

| Parameter         | Type                            | Default   | Description                                                                 |
|------------------|----------------------------------|-----------|-----------------------------------------------------------------------------|
| `vector`         | `List[float]`, `List[List[float]]`, or `np.ndarray`  | *required* | The query vector (single: `List[float]` or 1D `np.ndarray`) or batch of query vectors (`List[List[float]]` or 2D `np.ndarray`). Must match the index dimension and contain only finite values. |
| `filter`         | `Dict[str, Any] \| None`         | `None`    | Optional metadata filter. Values may be a plain value for equality or a dict of operators, and `$and`, `$or` and `$not` compose them. See [Filter Operators](#-filter-operators-reference) and [Boolean composition](#-boolean-composition). |
| `top_k`          | `int`                            | `10`      | Number of nearest neighbors to return, from 0 to 65,536. |
| `ef_search`      | `int \| None`                    | see below | Search complexity parameter, from 0 to 131,072. Higher values improve accuracy at the cost of speed. |
| `return_vector`  | `bool`                           | `False`   | If `True`, each result includes the stored embedding vector under a `vector` key. |
| `rerank`         | `int \| None`                    | derived from the record count | Candidates fetched per requested result before rescoring against raw vectors. Only applies to a quantized index whose `storage_mode` is `quantized_with_raw`. See [Quantized search accuracy](#-quantized-search-accuracy). |

**The default `ef_search` depends on the distance metric.** It is `max(2 × top_k, 100)` for `cosine` and `max(2 × top_k, 150)` for `l1` and `l2`.

A query vector containing `NaN` or an infinity raises `ValueError` rather than returning meaningless distances.

<br/>

### 🧰 Additional functionality

ZeusDB Vector Database includes a suite of utility functions to help you inspect, manage, and maintain your index. You can view index configuration, attach custom metadata, list stored records, and remove vectors by ID.

#### ☑️ Check the details of your HNSW index

```python
print(index.info())
```
*Output*
```
HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=10000, vectors=5, quantization=none)
```

The `vectors=` field is the live record count, in every storage mode. `get_vector_count()` returns the same number. `get_stats()["raw_vectors_stored"]` is the one that counts raw vectors specifically, and on a trained `quantized_only` index it is zero.

Other single-value accessors: `index.dim`, `index.space`, `index.m`, `index.ef_construction`, `index.expected_size`, `index.get_space()`, `len(index)`, `index.get_vector_count()`, `index.has_quantization()`, `index.can_use_quantization()`, and `VectorDatabase.available_index_types()`.

`index.dim`, `index.space`, `index.m`, `index.ef_construction` and `index.expected_size` are read-only properties. `get_space()` is the same value as a method and is kept for callers already using it.

```python
print(index.m, index.ef_construction, index.expected_size)
```
*Output*
```
16 200 10000
```

<br/>

#### ☑️ Add index level metadata

Index level metadata is a flat `str` to `str` map, separate from the per-record metadata used for filtering. It is preserved by `save()` and `load()`.

```python
index.add_metadata({
    "creator": "John Smith",
    "version": "0.1",
    "created_at": "2024-01-28T11:35:55Z",
    "embedding_model": "openai/text-embedding-ada-002",
    "environment": "production",
})

# View index level metadata by key
print(index.get_metadata("creator"))

# View all index level metadata
for key, value in sorted(index.get_all_metadata().items()):
    print(f"{key}: {value}")
```
*Output*
```
John Smith
created_at: 2024-01-28T11:35:55Z
creator: John Smith
embedding_model: openai/text-embedding-ada-002
environment: production
version: 0.1
```

`get_all_metadata()` returns a `dict` whose iteration order is not stable, which is why the example sorts it.

<br/>

#### ☑️ List records in the index

```python
for record_id, metadata in index.list(number=5):
    print(record_id, metadata)
```
*Output*
```
doc_001 {'author': 'Alice'}
doc_002 {'author': 'Bob'}
doc_003 {'author': 'Alice'}
doc_004 {'author': 'Bob'}
doc_005 {'author': 'Alice'}
```

`list()` returns `(id, metadata)` tuples **in the order the records were added**, and `offset` pages through them. It lists every record, in every storage mode, and the order survives `save()` and `load()`.

```python
print(index.list(number=2, offset=0))
print(index.list(number=2, offset=2))
print(index.list(number=2, offset=4))
print(index.list(number=2, offset=99))
```
*Output*
```
[('doc_001', {'author': 'Alice'}), ('doc_002', {'author': 'Bob'})]
[('doc_003', {'author': 'Alice'}), ('doc_004', {'author': 'Bob'})]
[('doc_005', {'author': 'Alice'})]
[]
```

An offset past the end returns an empty list rather than raising.

**Deleting while you page shifts the pages under `offset`.** Removing a record ahead of your cursor moves everything behind it up by one, so the next page skips one. Page with `after` instead, which names the last ID you saw.

```python
paged = vdb.create("hnsw", dim=2)
paged.add({"ids": [f"p{n}" for n in range(5)], "embeddings": [[n, 0.0] for n in range(5)]})

first = paged.list(number=2)
print([record_id for record_id, _ in first])
paged.remove_point("p0")
print([record_id for record_id, _ in paged.list(number=2, after=first[-1][0])])
print([record_id for record_id, _ in paged.list(number=2, offset=2)])
```

*Output*
```
['p0', 'p1']
['p2', 'p3']
['p3', 'p4']
```

`offset` skipped `p2` because a record ahead of it was removed. `after` did not, because it names a position rather than a count.

`after` and `offset` cannot both be given. If the record `after` names has itself been removed there is no position to resume from, and the call raises `KeyError` rather than returning a page from somewhere else.

<br/>

#### ☑️ Inspect index statistics

```python
stats = index.get_stats()
for key in ["total_vectors", "graph_nodes", "stranded_graph_nodes", "storage_mode_description"]:
    print(f"{key}: {stats[key]}")
```
*Output*
```
total_vectors: 5
graph_nodes: 5
stranded_graph_nodes: 0
storage_mode_description: raw_only
```

`get_stats()` returns a `str` to `str` map. Every key it carries:

| Key | Holds |
| --- | --- |
| `dimension`, `space`, `m`, `ef_construction`, `expected_size`, `index_type` | The configuration the index was created with |
| `total_vectors` | The live record count |
| `graph_nodes`, `stranded_graph_nodes` | Nodes in the HNSW graph, and how many of them no record uses |
| `raw_vectors_stored`, `quantized_codes_stored` | Records held at full width, and records held as codes |
| `storage_mode`, `storage_mode_description`, `storage_strategy` | What the index is storing and serving |
| `thread_safety` | The locking the index uses |
| `graph_memory_mb` | The HNSW graph, being the neighbour lists and, on a quantized index, the codes it scores against. It holds no raw vector |
| `raw_vectors_memory_mb` | The raw vectors, which are held once |
| `quantized_codes_memory_mb` | The codes, which grow with the record count |
| `codebook_memory_mb`, `sdc_table_memory_mb`, `centroid_norm_memory_mb` | The trained tables, fixed by `dim`, `subvectors` and `bits` |
| `index_bookkeeping_memory_mb` | The hash tables that find a record |
| `total_memory_mb` | The sum of the seven figures above |
| `quantization_type` | `pq`, or `none` on an unquantized index |
| `raw_vectors_retained` | The storage mode's policy, `none_once_trained` or `all_records`. Quantized indexes only |

On a quantized index it also carries `quantization_active`, `quantization_trained`, `quantization_compression_ratio`, `quantization_training_size`, `training_progress`, `training_threshold_reached`, `training_vectors_needed`, `raw_vectors_retained` and the rerank calibration keys below.

**`total_memory_mb` is what the index asked the allocator for, not what the process holds.** Either can be the larger. An arena reserved and not yet written is asked for and not resident, and the allocator's own bookkeeping is resident and not asked for. Measured on 50,000 real 1,536-dimensional embeddings, one index per interpreter, against the resident set delta across the build:

| mode | reported | resident | reported / resident |
|---|---:|---:|---:|
| no quantization | 347.90 MiB | 334.43 MiB | 1.04 |
| `quantized_with_raw` | 348.01 | 361.70 | 0.96 |
| `quantized_only` | 55.04 | 69.12 | 0.80 |

**Size infrastructure from the resident figure rather than from this one.** The gap is widest under `quantized_only`, where the fixed tables and the hash tables that find a record are most of what is left.

`index_bookkeeping_memory_mb` is proportional to the record count and independent of the dimension. It is not independent of the metadata, because the per-record metadata map is one of the tables it counts.

It also reports what a quantized search will fetch. `rerank_default_fetch` is the number of candidates a search at `top_k=10` fetches and rescores at the record count the index holds now, and a search at a larger `top_k` fetches more than it reports. On an index that does not rerank, being `quantized_only` or one not yet trained, it reads 10, the page itself. `rerank_calibrated` is `true` on a trained `quantized_with_raw` index and `false` on every other one, including an index saved before the calibration existed. When it is `true`, these report what training measured:

| Key | Holds |
| --- | --- |
| `rerank_calibration_fetch` | The fetch measured on the training sample |
| `rerank_calibration_records` | The records it was measured over |
| `rerank_calibration_queries` | The queries it used |
| `rerank_calibration_target_recall` | The recall it was measured to reach |
| `rerank_calibration_exponent` | How the fetch is scaled as the index grows |
| `rerank_calibration_fit_fetches` | The fetches the exponent was fitted from, comma separated |
| `rerank_calibration_pages` | The page sizes the fetch was measured at |
| `rerank_calibration_page_fetches` | The fetch at each of those pages |
| `rerank_calibration_page_exponent` | The slope through those pages |
| `rerank_calibration_ms` | What the calibration cost |

<br/>

#### ☑️ Remove Records

Remove a vector and its metadata with `.remove_point(id)`. This performs a <u>logical deletion</u>:
- The vector is deleted from internal storage.
- The metadata is removed.
- The vector ID is no longer returned by `.contains()`, `.get_records()`, or `.search()`.

```python
index.remove_point("doc_001")
print("doc_001 present:", index.contains("doc_001"))
print("records remaining:", index.get_vector_count())
```
*Output*
```
doc_001 present: False
records remaining: 4
```

**⚠️ Please Note:** Due to the nature of HNSW, the underlying graph node remains in memory after a point is removed. Searches never return it, but it still occupies memory and edge slots. `compact()` reclaims those nodes.

`remove_points(ids)` and `remove_where(filter)` remove a batch and a filtered set. Both are below, at the end of this section.

<br/>

#### ♻️ Reclaim space left by removals and overwrites

Both `remove_point()` and an overwriting `add()` leave a node behind in the graph. `compact()` rebuilds the graph in memory and returns the number of nodes it reclaimed. IDs, metadata, stored vectors, quantized codes and PQ training state all survive.

```python
print("stranded graph nodes:", index.get_stats()["stranded_graph_nodes"])
print("reclaimed:", index.compact())
print("stranded graph nodes:", index.get_stats()["stranded_graph_nodes"])
```
*Output*
```
stranded graph nodes: 1
reclaimed: 1
stranded graph nodes: 0
```

`compact()` costs a full rebuild, proportional to the number of live records rather than to the amount of debris, and it holds both graphs in memory while it runs. It returns 0 and does nothing when there is nothing to reclaim. It is never automatic, so schedule it when your workload has accumulated deletions.

<br/>

#### ♻️ Change `m` after the fact with `rebuild()`

`m` is chosen from `expected_size` when the index is created, so an index declared for far fewer records than it received runs at a degree meant for the smaller one. `rebuild(m=..., expected_size=..., ef_construction=...)` builds the graph again at a new configuration, in place, and every record keeps its vector, its metadata and its id. Pass any of the three.

```python
sized_wrong = vdb.create("hnsw", dim=8, expected_size=100, m=4)
sized_wrong.add({
    "ids": [f"v{i}" for i in range(400)],
    "embeddings": [[float(i % 7) + j * 0.1 for j in range(8)] for i in range(400)],
})
print(sized_wrong.m, sized_wrong.expected_size, len(sized_wrong))
print(sized_wrong.rebuild(m=16, expected_size=400))
print(sized_wrong.m, sized_wrong.expected_size, len(sized_wrong))
```

*Output*
```
4 100 400
400
16 400 400
```

It returns the node count of the graph it built, which is the live record count. Passing none of the three raises, because rebuilding the graph as it stands is `compact()`. An invalid value raises the message `create()` raises for it.

**Raise `m` where an index outgrew its declaration, and schedule it.** It costs a full rebuild, 27.0 seconds at 100,000 real 128 dimensional vectors, and it holds both graphs in memory while it runs. Nothing outside the graph is touched, so every filter returns what it returned and a save afterwards carries the new `m`.

<br/>

#### ☑️ Retrieve records by ID

Use `get_records()` to fetch one or more records by ID, with optional vector inclusion. It returns a list of dicts with `id`, `metadata`, and, when `return_vector` is true, `vector`.

```python
# Single record
print(index.get_records("doc_002", return_vector=False))

# Multiple records
print(index.get_records(["doc_002", "doc_003"], return_vector=False))

# Missing IDs are silently skipped
print(index.get_records(["doc_002", "missing_id"], return_vector=False))

# Vectors are included by default
record = index.get_records("doc_002")[0]
print(sorted(record.keys()), len(record["vector"]))
```

*Output*
```
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}]
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}, {'id': 'doc_003', 'metadata': {'author': 'Alice'}}]
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}]
['id', 'metadata', 'vector'] 8
```

⚠️ `get_records()` only returns results for IDs that exist in the index. Missing IDs are skipped by default, so a shorter list than you asked for is how a missing ID is reported, and the result does not say which one.

`strict=True` raises `KeyError` instead, naming every ID the index does not hold.

```python
try:
    index.get_records(["doc_002", "missing_id"], strict=True)
except KeyError as error:
    print(error)
```

*Output*
```
'get_records(strict=True) was asked for 1 id the index does not hold: missing_id. Call it without strict=True to receive the records that are present, or test an id with contains(id) first.'
```

**Under `cosine` the vector returned is the normalized one**, not the values you supplied, because that is what the index stores. Under `l1`, `l2` and `dot` it is what you supplied. There is no flag for this and no way to recover the original: the index keeps one copy of each vector and under `cosine` that copy is the unit vector. Keep your own copy if you need the values back.

**Under `quantized_only` the vector returned is a reconstruction from the record's code**, under the same `vector` key and with no marker saying so. Measured on 16 dimensional data with 4 subvectors and 8 bits, a reconstructed vector differed from the stored unit vector by 0.066 at the worst component. `get_stats()["raw_vectors_stored"]` is zero on such an index, which is how to tell. `quantized_with_raw` returns the stored vector exactly.

<br/>

#### ☑️ Count and test membership

`len(index)` is the live record count. `id in index` tests membership. `count(filter)` counts the records a metadata filter matches, and `count()` with no filter is `len(index)`.

```python
print(len(index))
print("doc_002" in index, "doc_001" in index)
print(index.count())
print(index.count({"author": "Alice"}))
print(index.count({"author": "Nobody"}))
```
*Output*
```
4
True False
4
2
0
```

`count()` is exact and therefore reads every record's metadata, so it costs what a filtered search costs. `contains(id)` is the same test as `in` and is kept for callers already using it.

<br/>

#### ☑️ Change a record's metadata

`update_metadata(id, metadata)` replaces one record's metadata without resupplying its vector. The record keeps its vector, its quantized codes and its graph node, and no node is stranded.

```python
print(index.get_records("doc_002", return_vector=False)[0]["metadata"])
print(index.update_metadata("doc_002", {"author": "Bob", "status": "reviewed"}))
print(sorted(index.get_records("doc_002", return_vector=False)[0]["metadata"].items()))
print(index.update_metadata("no_such_id", {"author": "Nobody"}))
print(index.get_stats()["stranded_graph_nodes"])
```
*Output*
```
{'author': 'Bob'}
True
[('author', 'Bob'), ('status', 'reviewed')]
False
0
```

The example sorts the second result because a record's metadata comes back as a `dict` whose key order is not stable between processes. Read metadata by key rather than by position.

**The replacement is wholesale, not a merge.** Any key you leave out is gone, which is what `add(overwrite=True)` already does. It returns `False` for an ID the index does not hold, and writes nothing in that case.

Use this rather than reading a record back with `get_records()` and adding it again. Measured at 20,000 records the round trip costs 486.5 microseconds against 1.57 for this, and it strands one graph node per update.

<br/>

#### ☑️ Remove several records at once

`remove_points(ids)` takes the lock once for the whole batch instead of once per ID. It returns the IDs that were **not** in the index, so an empty list means every one was removed. A repeated ID is removed on its first occurrence and is never reported missing.

```python
print(index.remove_points(["doc_004", "no_such_id"]))
print(len(index))
```
*Output*
```
['no_such_id']
3
```

`remove_where(filter)` removes every record a metadata filter matches, using the same filter language as `search()`, and returns how many it removed.

```python
print(index.remove_where({"author": "Alice"}))
print(len(index), index.count({"author": "Alice"}))
print(index.remove_where({"author": "Nobody"}))
```
*Output*
```
2
1 0
0
```

An unrecognised operator raises `ValueError` before any record is removed. A filter matching nothing removes nothing and returns `0`.

**`remove_where({})` is refused.** An empty filter matches every record everywhere else in this language, and here that would destroy the index. Name the records with `remove_points(ids)` if that is what you want, or use `clear()`.

Both leave one stranded graph node per record removed, exactly as `remove_point()` does, and neither calls `compact()`.

<br/>

#### ☑️ `delete()`, the shorter name for both

`delete(ids=...)` dispatches to `remove_points` and `delete(where=...)` to `remove_where`. Both of those stay.

```python
deletable = vdb.create("hnsw", dim=2, expected_size=10)
deletable.add({
    "ids": ["doc_1", "doc_2", "doc_3", "doc_4"],
    "embeddings": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]],
    "metadatas": [{"author": "Alice"}, {"author": "Bob"},
                  {"author": "Bob"}, {"author": "Alice"}],
})

print(deletable.delete(ids="doc_1"))
print(deletable.delete(ids=["doc_2", "no_such_id"]))
print(deletable.delete(where={"author": "Bob"}))
print(len(deletable))
```
*Output*
```
1
1
1
1
```

It returns **the number of records removed**, an `int`, whichever argument was given. `ids` takes a string or a list of strings. A repeated ID counts once. An ID that was not there counts zero rather than raising.

`remove_points` still returns the IDs it could not find, which is more than a count, so keep calling it where you need that.

**Passing both arguments raises, and so does passing neither.** Use `clear()` when emptying the index is what you mean.

<br/>

#### ☑️ Empty the index with `clear()`

`clear()` drops every record and returns how many went. It replaces the graph rather than removing records one at a time, so `stranded_graph_nodes` reads `0` afterwards.

```python
clearable = vdb.create("hnsw", dim=4, expected_size=10)
clearable.add({
    "ids": ["a", "b", "c", "d", "e"],
    "embeddings": [[1.0, 0, 0, 0], [0, 1.0, 0, 0], [0, 0, 1.0, 0],
                   [0, 0, 0, 1.0], [1.0, 1.0, 0, 0]],
})
clearable.remove_point("a")

print(clearable.clear())
print(len(clearable), clearable.get_stats()["stranded_graph_nodes"])
print(clearable.clear())
```
*Output*
```
4
0 0
0
```

**It keeps the index and drops the records.** `dim`, `space`, `m`, `ef_construction`, `expected_size`, the index level metadata and the quantization configuration all survive, and a fitted PQ codebook survives with them, so a trained quantized index can be refilled and searched without retraining. An index still collecting for training starts collecting again.

Clearing an empty index returns `0` and is not an error. The internal ID counter restarts, so the first generated ID after a clear is `vec_1` again.

<br/>

#### ♻️ Return the graph's spare capacity

An index built by inserting grows its graph buffers geometrically, so the last growth leaves the largest of them holding close to twice what they use. `shrink_to_fit()` returns that slack to the allocator and reports the bytes it released.

```python
fresh = vdb.create("hnsw", dim=8, expected_size=300)
fresh.add({
    "ids": [f"v{i}" for i in range(500)],
    "embeddings": [[float(i % 7) + j * 0.1 for j in range(8)] for i in range(500)],
})

before = float(fresh.get_stats()["graph_memory_mb"])
freed = fresh.shrink_to_fit()
after = float(fresh.get_stats()["graph_memory_mb"])
print(freed > 0, after < before)
print(fresh.shrink_to_fit())
```
*Output*
```
True True
0
```

The index above declared 300 records and was given 500, so its graph grew and left slack behind. `index`, the index used throughout this section, returns `0` instead, because `compact()` was called on it earlier and compaction already shrinks the graph it rebuilds.

**No node, edge or distance is touched**, so every search returns the same page with the same scores.

**Call it on an index that holds its records, not on one about to receive them.** On an empty index it hands back the whole creation reservation that `expected_size` bought, so every later insertion regrows the arenas from nothing.

**The index stays writable.** The buffers grow again on the next `add()`, which costs one reallocation. That is why it is never automatic.

<br/>

#### ☑️ Quantization status and performance reporting

Five further accessors report state that `get_stats()` also carries.

| Method | Returns |
| --- | --- |
| `is_training_ready()` | Whether the training threshold has been reached. `False` on an index with no quantization configuration |
| `training_vectors_needed()` | Records still to collect before training triggers. `0` on an index with no quantization configuration |
| `rebuild_with_quantization()` | Rebuilds the graph against the quantized codes and returns whether it did. Training and `load()` already do this, so calling it is normally redundant |
| `get_performance_info()` | A `str` to `str` map describing the search and insertion paths |
| `benchmark_concurrent_reads(query_count, max_threads)` | Times sequential against threaded searches over random queries, returning `sequential_qps`, `parallel_qps`, `speedup`, `sequential_time`, `parallel_time` and `threads_used` |

```python
ready = vdb.create("hnsw", dim=8, expected_size=1200, quantization_config={
    "type": "pq", "subvectors": 8, "bits": 8, "training_size": 1000,
})
print(ready.is_training_ready(), ready.training_vectors_needed())
print(sorted(ready.get_performance_info()))
```
*Output*
```
False 1000
['benefits', 'insertion_path', 'quantization_accuracy_impact', 'quantization_compression', 'search_bottleneck', 'search_speedup_expected']
```

<br />

## 🗜️ Product Quantization

Product Quantization (PQ) is a vector compression technique that reduces memory usage by dividing each vector into subvectors and quantizing them independently. A record's compressed form is one byte per subvector, whatever the dimension, so an index over 1536-dimensional vectors with 8 subvectors stores 8 bytes per code in place of 6144 bytes of float32.

ZeusDB Vector Database's PQ implementation features:

✅ Automatic training, triggered on the `add()` call that reaches the configured threshold

✅ Compact codes, one byte per subvector per record

✅ Asymmetric Distance Computation (ADC) for fast search against the codes

✅ Automatic switch from raw to quantized storage once training completes

Compression is not free, and the accuracy cost is much larger than the memory saving suggests. Read [Quantized search accuracy](#-quantized-search-accuracy) before choosing a storage mode.

<br />

### 📘 Quantization Configuration Parameters

To enable PQ, pass a `quantization_config` dictionary to the `.create()` index method:

| Parameter | Type | Description | Valid Range | Default |
|-----------|------|-------------|-------------|---------|
| `type` | `str` | Quantization algorithm type | `"pq"` | *required* |
| `subvectors` | `int` | Number of vector subspaces. Must divide `dim` evenly | 1 to `dim` | derived from `dim`, see below |
| `bits` | `int` | Bits per quantized code, which sets the centroids per subvector to 2^bits | 1 to 8 | `8` |
| `training_size` | `int` | Records collected before training is triggered | ≥ 1000 | `10000` |
| `max_training_vectors` | `int \| None` | Maximum records used during training | ≥ `training_size` | `None` |
| `storage_mode` | `str` | `"quantized_only"` or `"quantized_with_raw"` | see below | `"quantized_only"` |

**Compression ratio is `dim × 4 / subvectors`.** More subvectors means a longer code, so it lowers the compression ratio and raises accuracy. Fewer subvectors means the opposite. At `dim=1536`, 8 subvectors gives 768x and 16 subvectors gives 384x.

**`subvectors` defaults to `dim / 32`, clamped to between 8 and 192, snapped to a divisor of `dim`.** That holds the compression ratio at 128x, which is the quantity accuracy follows.

| `dim` | default `subvectors` | compression |
|-------|----------------------|-------------|
| 64 | 8 | 32x |
| 128 | 8 | 64x |
| 256 | 8 | 128x |
| 768 | 24 | 128x |
| 1536 | 48 | 128x |
| 3072 | 96 | 128x |

Going lower than 128x costs memory and build time and buys nothing on recall until the ratio reaches about 16x, where the fetch collapses and query time falls instead. See [Quantized search accuracy](#-quantized-search-accuracy). Below `dim=256` the floor of 8 subvectors binds. Pass `subvectors` explicitly for a cheaper, less accurate setting.

`bits` does not change the size of a record's code, which is always one byte per subvector, so lowering it saves no memory per record. It sets the number of centroids per subvector to 2^bits, which sizes the codebook and the centroid distance table. Lowering it costs recall and shortens the build. Leave it at 8 unless the fixed cost or the build time is the constraint.

`create()` emits a `UserWarning` when the configuration looks unbalanced, for example when the compression ratio exceeds 50x, and another when `storage_mode` is `quantized_with_raw`. The ratio warning does not fire on a `subvectors` the library derived, only on one you passed.

It also warns when `quantized_only` cannot repay its fixed memory at the `expected_size` you declared, naming the record count above which it starts saving. Raise `expected_size` if your estimate was low, or drop `quantization_config`. `quantized_with_raw` has no such record count, because it holds more than an unquantized index at every one, and the warning that names the mode says so. A separate warning fires when `expected_size` is below `training_size`, because an index that never reaches its training threshold never trains.

<br/>

### 🔧 Usage Example 1

```python
from zeusdb_vector_database import VectorDatabase
import numpy as np

vdb = VectorDatabase()

quantization_config = {
    "type": "pq",                        # `pq` for Product Quantization
    "subvectors": 8,                     # 8 subvectors of 192 dims each
    "bits": 8,                           # 256 centroids per subvector (2^8)
    "training_size": 1000,               # Train once 1,000 records are collected
    "storage_mode": "quantized_with_raw" # Keep raw vectors so results can be reranked
}

index = vdb.create(
    index_type="hnsw",
    dim=1536,                                # OpenAI `text-embedding-3-small` dimension
    expected_size=2500,
    quantization_config=quantization_config
)

# Add vectors. Training triggers automatically at the threshold.
rng = np.random.default_rng(0)
documents = {
    "ids": [f"doc_{i}" for i in range(2500)],
    "embeddings": rng.random((2500, 1536), dtype=np.float32),
    "metadatas": [{"category": "tech", "year": 2026} for _ in range(2500)],
}

result = index.add(documents)
print("inserted:", result.total_inserted)

# Check quantization status
print("training progress:", f"{index.get_training_progress():.1f}%")
print("storage mode:", index.get_storage_mode())
print("is quantized:", index.is_quantized())

# Get compression statistics
quant_info = index.get_quantization_info()
print("compression ratio:", f"{quant_info['compression_ratio']:.1f}x")
print("codebook memory:", f"{quant_info['memory_mb']:.1f} MB")

# Search works the same way on a quantized index
query_vector = rng.random(1536, dtype=np.float32)
results = index.search(vector=query_vector, top_k=3)
print("results:", len(results), "| keys:", sorted(results[0].keys()))
```

*Output*
```
inserted: 2500
training progress: 100.0%
storage mode: quantized_active
is quantized: True
compression ratio: 768.0x
codebook memory: 1.5 MB
results: 3 | keys: ['id', 'metadata', 'score']
```

The result IDs and scores depend on the data, so they are not shown. Production indexes use a much larger `training_size`; 1,000 is the minimum the validator accepts and keeps this example quick.

`index.info()` reports the quantization state as well:

```python
print(index.info())
```

*Output*
```
HNSWIndex(dim=1536, space=cosine, m=16, ef_construction=200, expected_size=2500, vectors=2500, quantization=pq(subvectors=8, bits=8, trained, active, compression=768.0x))
```

<br />

### 🔧 Usage Example 2 - with explicit storage mode

```python
from zeusdb_vector_database import VectorDatabase

vdb = VectorDatabase()

quantization_config = {
    "type": "pq",
    "subvectors": 8,
    "bits": 8,
    "training_size": 10000,
    "max_training_vectors": 50000,
    "storage_mode": "quantized_only"    # Drop raw vectors once training completes
}

index = vdb.create(
    index_type="hnsw",
    dim=3072,                           # OpenAI `text-embedding-3-large` dimension
    expected_size=100000,
    quantization_config=quantization_config
)
```

<br/>

### 📦 Storage modes

| Mode | What it stores | Rerank available | Memory |
|------|----------------|------------------|--------|
| `quantized_only` | Codes for every record; the raw vectors collected for training are released when training completes | No | Lowest of the three |
| `quantized_with_raw` | Codes and raw vectors for every record | Yes | Highest of the three. It adds the codes and the trained tables to everything an unquantized index holds |

Two consequences of `quantized_only` are worth knowing before you pick it.

**The training records are held at full width only until training completes.** Records collected before the threshold is reached are stored raw so the quantizer has something to train on. The moment training completes they are encoded and their raw copies are released, so a trained index in this mode holds no raw vector for any record.

**Once training completes every record exists only as a code, so the vector you read back is an approximation.** Every accessor sees every record. `get_records(..., return_vector=True)` and `search(..., return_vector=True)` reconstruct the vector from its code. Only `quantized_with_raw` reads back exactly, and `get_stats()["raw_vectors_stored"]` reading zero is how you confirm the release happened.

```python
only = vdb.create("hnsw", dim=1536, expected_size=2500, quantization_config={
    "type": "pq",
    "subvectors": 8,
    "bits": 8,
    "training_size": 1000,
    "storage_mode": "quantized_only",
})
only.add(documents)   # the same 2,500 records used in Usage Example 1

print("storage mode:", only.get_storage_mode())
stats = only.get_stats()
print("raw vectors kept:", stats["raw_vectors_stored"])
print("quantized codes:", stats["quantized_codes_stored"])
print("records:", only.get_vector_count())
print("contains doc_0 (added before training):", only.contains("doc_0"))
print("contains doc_2000 (added after training):", only.contains("doc_2000"))
print("get_records doc_2000 returns:", len(only.get_records("doc_2000")), "record")
```

*Output*
```
storage mode: quantized_active
raw vectors kept: 0
quantized codes: 2500
records: 2500
contains doc_0 (added before training): True
contains doc_2000 (added after training): True
get_records doc_2000 returns: 1 record
```

**`quantized_only` is the memory mode and `quantized_with_raw` is the accuracy mode.** A raw vector is held once, in a store the graph is handed. `quantized_only` replaces it with a code and holds a second code in the map that finds a record by id, so it saves `dim × 4 - 2 × subvectors` bytes per record against a codebook and a centroid distance table it holds whatever the record count. `quantized_with_raw` keeps the raw vector and adds both codes and both tables to it, so it holds more than an unquantized index at every record count.

Measured resident, one index per interpreter, 50,000 records of real embeddings in each mode over the same data:

| dataset | dim | unquantized | `quantized_only` | `quantized_with_raw` |
|---|---:|---:|---:|---:|
| dbpedia-openai | 1,536 | 334.4 MiB | 69.1 MiB, 0.21x | 361.7 MiB, 1.08x |
| sift-128 | 128 | 66.6 MiB | 50.5 MiB, 0.76x | 76.1 MiB, 1.14x |

**Pick `quantized_only` when memory is the constraint and `quantized_with_raw` when accuracy is.** What `quantized_only` saves is set by the share of a record that is the vector, and that share falls with the dimension: 37% at `dim=128` and 88% at `dim=1,536` on the rows above. `get_stats()` prices your own index on your own records, which is the figure to size against.

`create()` warns when `quantized_only` cannot repay its fixed tables at the `expected_size` you declared, naming the record count above which it starts saving. Raise `expected_size` if your estimate was low, or drop `quantization_config`. `quantized_with_raw` never repays them, so it gets the warning that names the mode instead.

<br/>

### 🎯 Quantized search accuracy

**Quantized search is far less accurate than raw search, and `quantized_only` cannot be repaired by tuning.** ADC scores candidates against the codes, and a code discards most of the information in a vector. Rerank fixes this by over-fetching candidates and rescoring them against raw vectors, which is only possible when the raw vectors are still there.

Measured on 6,000 clustered 128-dimensional vectors with 8 subvectors and 8 bits, recall at 10 against exact cosine search:

| Configuration | Recall@10 |
|---------------|-----------|
| No quantization | 1.00 |
| `quantized_only` | 0.16 |
| `quantized_with_raw`, `rerank=0` | 0.15 |
| `quantized_with_raw`, default rerank | 1.00 |

The exact figures depend on your data, but the shape does not. If you need quantization and you need accuracy, use `quantized_with_raw` and leave rerank on.

**A quantized `cosine` index ranks by the cosine distance to the reconstruction, and reports it.** A reconstruction is assembled from independently trained per-subspace centroids and nothing renormalises it, so it is not a unit vector even where the record it stands for was. Measured on 25,000 OpenAI `text-embedding-ada-002` vectors at `dim=1,536` with 48 subvectors and 8 bits, reconstructed norms ran 0.85 to 0.96 against a stored norm of 1.0.

That matters because a quantized graph works from a table of squared L2 distances, and on those reconstructions the squared L2 ran at about 1.86 times the cosine distance rather than at exactly twice it. The gap is each record's own reconstruction length, which the index recovers from the codes, so the score you get back is the cosine distance and not a multiple of it. Ranking by cosine rather than by squared L2 also moves the page, and it measured better on every corpus and subvector count tried, by 0.0015 to 0.0518 of recall at 10 over 40,000 held-out queries each.

**How deep a search has to fetch to hold that recall depends on your data, not on the record count.** Measured on three real datasets at 100,000 records with the default `subvectors`, the fetch that reaches mean recall at 10 of 0.99:

| dataset | dim | compression | fetch for 0.99 | share of corpus |
|---|---:|---:|---:|---:|
| dbpedia-openai (ada-002) | 1,536 | 128x | 494 | 0.49% |
| sift-128 | 128 | 64x | 426 | 0.43% |
| glove-100 | 100 | 40x | 5,143 | 5.14% |

No formula in the record count fits those three, so ZeusDB measures the fetch on your data instead. A `quantized_with_raw` index measures it when training completes, and scales what it measured with the record count and with the page size you ask for. `get_stats()["rerank_default_fetch"]` reports what a search at `top_k=10` will fetch on the index as it stands.

**On data with no resolvable structure no fetch works.** Once the group the codes cannot separate is smaller than `top_k`, the true top ten span groups and nothing reaches them. Measure recall on your own data before you rely on quantization.

What `rerank` does:

| `rerank` | Effect |
| --- | --- |
| omitted | Uses the calibrated fetch. It is the only setting that holds recall across corpus sizes and across datasets |
| `N` of 1 or more | Fetches `top_k × N` candidates, a fixed multiple of the page that does not move with the corpus. Use it to override the default deliberately |
| `0` | Turns reranking off and returns the ADC scores and ordering |

A page below ten fetches what a page of ten fetches, so pass `rerank` explicitly if you want a shallower page to cost less. `rerank` has no effect on an unquantized index or on a `quantized_only` one, and both ignore it. With rerank on the scores you get back are raw-vector distances, and with it off they are distances to the reconstruction. Both are on the scale the index's own space reports, so a page is on one scale whichever you asked for.

An index trained before the calibration existed, and any index loaded from a directory saved by one, carries no calibration and falls back to a fixed fetch of 2% of the record count. `get_stats()["rerank_calibrated"]` reads `false` for it. Rebuild the index to calibrate it.

**Above roughly 10,000 records a reranked quantized search is slower than an unquantized one, and the gap widens as the index grows.** That is the price of the default holding recall. On dbpedia-openai at `dim=1,536`, paired against an unquantized index over the same records, 200 queries one each in turn:

| records | calibrated fetch | unquantized | quantized, default rerank | ratio |
|---:|---:|---:|---:|---:|
| 10,000 | 277 | 0.75 ms | 0.71 ms | 0.95 |
| 25,000 | 411 | 0.79 ms | 0.97 ms | 1.23 |
| 50,000 | 554 | 1.17 ms | 1.54 ms | 1.32 |
| 100,000 | 747 | 1.18 ms | 2.12 ms | 1.79 |

Each row is one process building both indexes over the same records, so the ratio is the figure to read. Quantization remains a memory decision that costs query time. Lower `rerank` explicitly if query time matters more to you than recall, and measure what it costs you.

**`ef_search` does nothing on a reranked quantized search.** The graph traversal widens to the number of candidates the fetch asks for, which is already wider than any `ef_search` a caller is likely to set, and setting it smaller is discarded. Change `rerank` instead. On an unquantized search, and on a quantized search with `rerank=0`, `ef_search` applies normally.

### 📊 Performance Characteristics

- **Training**: happens once, on the `add()` call that reaches `training_size`. That call takes noticeably longer than the others. On `quantized_with_raw` it also calibrates the rerank fetch, which `get_stats()["rerank_calibration_ms"]` prices.
- **Memory**: a record's code is `subvectors` bytes against `dim × 4` for a raw vector, and a raw vector is held once. `quantized_only` saves and `quantized_with_raw` costs. The table in Storage modes prices both at `dim=128` and `dim=1,536`.
- **Search speed**: an unreranked quantized search is faster than a raw search. A reranked one is slower above roughly 10,000 records, and the table above prices it.
- **Build speed**: a quantized build is faster than an unquantized one, and it slows as `subvectors` rises. At 100,000 records of `dim=768` it is 137 s against 231 s at the default `subvectors`.
- **Accuracy**: see the tables above. Treat quantization as a memory decision that costs accuracy and query time, not as a free win.

<br/>

## 💾 Persistence

ZeusDB Vector Database can save and restore complete indexes on disk, which lets you preserve your work, move indexes between systems, and back up production deployments.

The persistence system supports:

✅ **Complete state preservation** for vectors, per-record metadata, index level metadata, ID mappings and quantization models
✅ **Hybrid storage format**, binary encoding for vectors with human-readable JSON for metadata
✅ **Quantization support**, both raw and quantized storage modes, including the trained codebook
✅ **Training state recovery**, so an index saved mid-collection resumes collecting
✅ **Format versioning**, so a directory this build cannot interpret is refused rather than misread
✅ **Atomic saves**, so a reader sees the whole previous index or the whole new one
✅ **A digest per artefact**, checked on load, so a file that has changed since it was written is refused

<br/>

### 💾 Saving an Index - .save()

Use the `.save()` method to persist your index to a `.zdb` directory:

```python
from zeusdb_vector_database import VectorDatabase
import numpy as np
import os

vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536, space="cosine", expected_size=1000)

rng = np.random.default_rng(1)
vectors = rng.random((1000, 1536), dtype=np.float32)
index.add({
    "ids": [f"doc_{i}" for i in range(1000)],
    "embeddings": vectors,
    "metadatas": [{"category": f"cat_{i % 5}", "index": i} for i in range(1000)],
})

index.save("my_index.zdb")
print("saved:", sorted(os.listdir("my_index.zdb")))
```

*Output*
```
saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
```

<br />

### 📂 Loading an Index - .load()

Use the `.load()` method to restore a previously saved index:

```python
vdb = VectorDatabase()
loaded_index = vdb.load("my_index.zdb")

print("vectors:", loaded_index.get_vector_count())
print(loaded_index.info())

results = loaded_index.search(vectors[0].tolist(), top_k=3)
print("top hit:", results[0]["id"])
```

*Output*
```
vectors: 1000
HNSWIndex(dim=1536, space=cosine, m=16, ef_construction=200, expected_size=1000, vectors=1000, quantization=none)
top hit: doc_0
```

**Loading reads the saved graph back rather than rebuilding it**, so a reloaded index returns the same result pages as the index that was saved, with the same IDs and the same scores. Load time is proportional to the size of the directory rather than to the cost of building the index: 50,000 records at 1,536 dimensions load in 1.1 seconds against a 156 second build.

The graph is rebuilt by re-inserting every record only when the saved graph cannot be used, which covers a directory whose graph files were lost or damaged and one written by a release too old for this build to interpret. Set `ZEUSDB_LOAD_REBUILD_GRAPH=1` to ask for that rebuild on a directory whose graph is perfectly readable, which is how an index built by an earlier release picks up graph improvements made since.

<br />

### 🗜️ Persistence with Product Quantization

A quantized index comes back quantized, with its codebook and training state intact:

```python
quantization_config = {
    "type": "pq",
    "subvectors": 8,
    "bits": 8,
    "training_size": 1000,
    "storage_mode": "quantized_with_raw",
}

vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536, expected_size=2000,
                   quantization_config=quantization_config)

rng = np.random.default_rng(2)
index.add({
    "ids": [f"vec_{i}" for i in range(2000)],
    "embeddings": rng.random((2000, 1536), dtype=np.float32),
})

print("quantization active:", index.is_quantized())
index.save("quantized_index.zdb")

loaded_index = vdb.load("quantized_index.zdb")
print("quantization active after load:", loaded_index.is_quantized())
print("storage mode after load:", loaded_index.get_storage_mode())
print("saved:", sorted(os.listdir("quantized_index.zdb")))
```

*Output*
```
quantization active: True
quantization active after load: True
storage mode after load: quantized_active
saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
```

<br/>

### 📁 Index Directory Structure
The `.save()` method creates a directory containing all index components:

```
my_index.zdb/
├── manifest.json           # Index metadata and file inventory
├── config.json             # HNSW configuration and index level metadata
├── mappings.bin            # ID mappings (binary format)
├── metadata.json           # Per-record metadata (JSON format)
├── vectors.bin             # Raw vectors (whenever the index holds any)
├── quantization.json       # PQ configuration (if enabled)
├── pq_centroids.bin        # Trained centroids (if PQ trained)
├── pq_codes.bin            # Quantized codes (if PQ active)
└── hnsw_index.zdbgraph     # HNSW graph structure and payload
```

`manifest.json` lists every file the save wrote under `files_included` and is the last file written, so it is the inventory of what the directory does hold. Beside the list, `file_digests` records each artefact's length and a digest of its contents.

A directory saved by 0.6.0 or earlier holds `hnsw_index.hnsw.graph` and `hnsw_index.hnsw.data` in place of `hnsw_index.zdbgraph`. Opening it still works: the graph is rebuilt once from the stored records, and the next `.save()` writes the single file.

**`load()` refuses a directory that does not hold what its manifest names.** It checks `files_included` before it reads anything, and the graph dump is the one exempt artefact, because every record carries what the graph is built from. A directory missing any other file will not open, and the refusal names the file and says what it held. Restore it from a copy; the missing file cannot be rebuilt from the ones that remain. A file the manifest does not name is neither read nor complained about.

**It also refuses a file that is present and has changed.** Each artefact is checked against the length and digest `file_digests` records for it, before anything parses it, so a file edited in place is refused with its name and both digests in the message. The graph dump carries its own header and payload checksums instead, so the manifest records only its length; a dump that disagrees is rebuilt rather than refused.

A directory saved before 0.8.0 carries no digests, so nothing is verified and it loads exactly as it did.

A file that is present and does not parse is a different failure with a different message, of the form `Failed to parse config.json` or `Failed to deserialize mappings.bin`.

<br/>

### 🔄 Complete Save/Load Workflow
A full persistence lifecycle with integrity checks:

```python
from zeusdb_vector_database import VectorDatabase
import numpy as np

# === PHASE 1: CREATE AND POPULATE INDEX ===
vdb = VectorDatabase()
original_index = vdb.create("hnsw", dim=1536, space="cosine", expected_size=500)

rng = np.random.default_rng(42)
vectors = rng.random((500, 1536), dtype=np.float32)

original_index.add({
    "ids": [f"doc_{i:03d}" for i in range(500)],
    "embeddings": vectors,
    "metadatas": [
        {
            "category": ["science", "tech", "health", "finance"][i % 4],
            "priority": i % 10,
            "published": i % 2 == 0,
            "tags": ["important", "featured"] if i % 5 == 0 else ["standard"],
        }
        for i in range(500)
    ],
})

original_index.add_metadata({
    "dataset": "demo_collection",
    "created_by": "data_team",
    "version": "1.0",
})

query_vector = vectors[0].tolist()
original_results = original_index.search(query_vector, top_k=3)

# === PHASE 2: SAVE, THEN LOAD ===
original_index.save("demo_index.zdb")
loaded_index = vdb.load("demo_index.zdb")

# === PHASE 3: VERIFY INTEGRITY ===
assert loaded_index.get_vector_count() == original_index.get_vector_count()
assert loaded_index.info() == original_index.info()
assert loaded_index.get_all_metadata() == original_index.get_all_metadata()

loaded_results = loaded_index.search(query_vector, top_k=3)
assert [r["id"] for r in loaded_results] == [r["id"] for r in original_results]

filtered = loaded_index.search(
    query_vector,
    filter={"category": "science", "published": True},
    top_k=20,
)

print("records:", loaded_index.get_vector_count())
print("index metadata fields:", len(loaded_index.get_all_metadata()))
print("filtered hits:", len(filtered))
print("all checks passed")
```

*Output*
```
records: 500
index metadata fields: 3
filtered hits: 20
all checks passed
```

### ⚠️ Important Notes on Persistence

- **Directory, not a file.** `.save()` creates a directory. You need write permission for the target location.

- **Atomic.** A save writes `<name>.zdbtmp` beside the target and renames it into place, so a reader sees the previous index or the new one and never a mixture. An interrupted save leaves the previous directory intact and loadable, and the staging directory is removed.

  Replacing an existing directory takes two renames rather than one, because neither Windows nor POSIX can rename a directory over a non-empty one: the target moves to `<name>.zdbold`, the new directory moves in, then `<name>.zdbold` is removed. Between the two renames the target does not exist. A process killed in that window leaves the whole previous index at `<name>.zdbold`, and the next save moves it back.

- **Overwriting is clean.** The new directory is built from nothing, so an artefact from an earlier save cannot survive. Saving a plain index over a quantized one leaves no `quantization.json`, `pq_centroids.bin` or `pq_codes.bin` behind.

- **Same volume.** The staging directory is a sibling of the target, so both are on the target's volume and the move is a rename rather than a copy.

- **Version compatibility.** The manifest records a format version. This build writes 1.1.0 and reads any 1.x. A different major version is refused.

- **Integrity checks on load.** Four run, in this order: the format version, then `files_included` against the directory, then each artefact against its recorded length and digest, then the restored record count against the count in `config.json`.

- **`save()` and `load()` are silent.** Every step they used to print to stdout is a `debug` log line instead, so a library caller sees nothing on stdout. Set `ZEUSDB_LOG_LEVEL=debug` to see the steps.

<br />

## 🏷️ Metadata Filtering

ZeusDB supports rich metadata with full type fidelity. Your metadata preserves the original Python data types, so integers stay integers and floats stay floats.

### 📘 Supported Types

| Type | Python Example | Notes |
|------|----------------|-------|
| **String** | `"Alice"` | Text data, IDs, categories |
| **Integer** | `42`, `2024` | Counts, years, IDs |
| **Float** | `4.5`, `29.99` | Ratings, prices, scores |
| **Boolean** | `True`, `False` | Flags, status indicators |
| **Null** | `None` | Missing or empty values |
| **Array** | `["ai", "science"]` | Tags, categories, lists |
| **Nested Object** | `{"key": "value"}` | Structured data |

Integers and floats compare by magnitude, so a stored integer `10` matches `{"eq": 10.0}` and `{"gte": 10.0}` alike. Booleans and strings do not cross into numbers.

<br/>

### 📘 Filter Operators Reference

A filter is a dict whose keys are field names, and all of them must hold. A field maps either to a plain value, which means equality, or to a dict of operators, all of which must hold. Three reserved keys, `$and`, `$or` and `$not`, compose whole filters rather than naming a field; see [Boolean composition](#-boolean-composition).

| Operator | Usage | Example | Description |
|----------|-------|---------|-------------|
| **Direct equality** | `{"field": value}` | `{"author": "Alice"}` | Equality for strings, numbers, booleans, null and arrays |
| `eq` | `{"eq": value}` | `{"source": {"eq": {"kind": "web"}}}` | Equality, including for nested objects |
| `ne` | `{"ne": value}` | `{"author": {"ne": "Alice"}}` | Not equal |
| `gt` | `{"gt": value}` | `{"rating": {"gt": 4.0}}` | Greater than (numeric) |
| `gte` | `{"gte": value}` | `{"year": {"gte": 2024}}` | Greater than or equal (numeric) |
| `lt` | `{"lt": value}` | `{"price": {"lt": 30}}` | Less than (numeric) |
| `lte` | `{"lte": value}` | `{"pages": {"lte": 100}}` | Less than or equal (numeric) |
| `contains` | `{"contains": value}` | `{"tags": {"contains": "ai"}}` | String contains substring, or array contains value |
| `startswith` | `{"startswith": value}` | `{"title": {"startswith": "The"}}` | String starts with substring |
| `endswith` | `{"endswith": value}` | `{"file": {"endswith": ".pdf"}}` | String ends with substring |
| `in` | `{"in": [values]}` | `{"lang": {"in": ["en", "es"]}}` | Value is in the provided array |
| `nin` | `{"nin": [values]}` | `{"lang": {"nin": ["en", "es"]}}` | Value is not in the provided array |
| `any` | `{"any": [values]}` | `{"tags": {"any": ["ai", "ml"]}}` | Array field shares at least one element with the provided array |
| `all` | `{"all": [values]}` | `{"tags": {"all": ["ai", "ml"]}}` | Array field holds every element of the provided array |
| `exists` | `{"exists": bool}` | `{"lang": {"exists": False}}` | Whether the record carries the field at all |
| `is_missing` | `{"is_missing": bool}` | `{"lang": {"is_missing": True}}` | The complement of `exists` |
| `is_null` | `{"is_null": bool}` | `{"lang": {"is_null": True}}` | The record carries the field and its value is null |

`any` and `all` exist because a field maps to one condition object, so it cannot carry `contains` twice. They ask their question of one field's array, where `$or` and `$and` compose whole filters across fields. On a field holding a plain value rather than an array, both read it as an array of one.

Three behaviours are worth knowing.

**A record that lacks the field never matches, whatever the operator.** That includes `ne` and `nin`. `{"lang": {"ne": "en"}}` and `{"lang": {"nin": ["en"]}}` do not match a record with no `lang` at all, and they agree because `nin` against a one-element array means what `ne` means.

**`exists`, `is_missing` and `is_null` are the three that ask about the field itself**, so they are the exception to the rule above and each is decided before the value is looked up. A missing field and a stored null are different: `{"lang": None}` stores a null and `{"lang": {"exists": False}}` matches only a record with no `lang` key.

```python
from zeusdb_vector_database import VectorDatabase

selector = VectorDatabase().create("hnsw", dim=2)
selector.add([
    {"id": "has", "values": [1.0, 0.0], "metadata": {"lang": "en"}},
    {"id": "null", "values": [0.0, 1.0], "metadata": {"lang": None}},
    {"id": "none", "values": [1.0, 1.0], "metadata": {}},
])
found = lambda f: sorted(r["id"] for r in selector.search([1.0, 0.0], filter=f, top_k=9))
print(found({"lang": {"exists": True}}))
print(found({"lang": {"is_missing": True}}))
print(found({"lang": {"is_null": True}}))
print(found({"lang": {"exists": True, "is_null": False}}))
```

*Output*
```
['has', 'null']
['none']
['null']
['has']
```

Each takes `True` or `False` and anything else raises. `is_null: False` is the complement of `is_null: True`, so it matches a record with no field at all; write "present and not null" as the conjunction above.

**A dict value is always read as operators.** Direct equality against a nested object has no plain form, because the two would be indistinguishable, so write it as `{"source": {"eq": {"kind": "web"}}}`. Writing `{"source": {"kind": "web"}}` raises `ValueError: Unknown filter operation: kind`.

**An unrecognised operator raises `ValueError` before the search runs**, rather than quietly matching nothing.

<br/>

### 🧩 Boolean composition

A filter is a conjunction of its keys. Three reserved keys compose whole filters instead of naming a field.

| Key | Takes | Means |
|-----|-------|-------|
| `$and` | a list of filters | every one of them holds |
| `$or` | a list of filters | at least one of them holds |
| `$not` | one filter | that filter does not hold |

**Precedence.** There is none to remember, because the structure is explicit. A mapping is an AND of everything in it, fields and groups alike, so `{"a": 1, "$or": [...]}` means `a == 1` AND the disjunction. A group's branches are each a whole filter, so a branch carrying two fields conjoins them.

**Nesting.** Groups nest to 10 levels, counting the filter itself as level one. A filter deeper than that raises `ValueError`.

**Reserved keys.** Exactly `$and`, `$or` and `$not`. The `$` prefix is not reserved, so a field named `$price` still filters. A field literally named `$or`, `$and` or `$not` cannot be filtered on, and a filter naming it raises.

**The empty cases.** `{"$and": []}` matches every record and `{"$or": []}` matches none, which is what `all` and `any` already do with an empty array.

```python
from zeusdb_vector_database import VectorDatabase

vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, space="l2")
index.add([
    {"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
     "metadata": {"lang": "en", "tier": "gold", "year": 2024}},
    {"id": "d2", "values": [0.2, 0.2, 0.2, 0.2],
     "metadata": {"lang": "es", "tier": "free", "year": 2023}},
    {"id": "d3", "values": [0.3, 0.3, 0.3, 0.3],
     "metadata": {"lang": "fr", "tier": "gold", "year": 2026}},
    {"id": "d4", "values": [0.4, 0.4, 0.4, 0.4],
     "metadata": {"tier": "free", "year": 2025}},
])
q = [0.1, 0.1, 0.1, 0.1]


def matched(filter):
    return sorted(hit["id"] for hit in index.search(vector=q, filter=filter, top_k=10))


# Either language. A flat filter cannot ask this, because one field maps to one
# condition and two conditions on it are conjoined.
print(matched({"$or": [{"lang": "en"}, {"lang": "es"}]}))

# Gold tier, and either recent or English. A branch is a whole filter, so the
# second one carries two fields and conjoins them.
print(matched({"tier": "gold",
               "$or": [{"year": {"gte": 2026}}, {"lang": "en"}]}))

# Not free tier. This is what `ne` does here, since every record has a tier.
print(matched({"$not": {"tier": "free"}}))

# Records with no lang field at all, which no operator can select.
print(matched({"$not": {"lang": {"all": []}}}))

# None of these, which is Qdrant's must_not over a group.
print(matched({"$not": {"$or": [{"lang": "es"}, {"tier": "gold"}]}}))
```

*Output*
```
['d1', 'd2']
['d1', 'd3']
['d1', 'd3']
['d4']
['d4']
```

<br/>

### ⏱️ What a filtered search costs

**A filter over a field left out of `indexed_fields` reads every record's metadata**, so it costs a great deal more than an unfiltered search. Declaring the field builds a column and removes that cost, which the next section measures.

Two paths serve a filter and the index chooses between them per search. At or below 5,000 matching records it scores every record that matched and ranks them, which is exact. Above 5,000 the graph traversal runs instead with the filter tested at every node it reaches, and recall there is the graph's own, measured at 0.96 and above on three real 100,000 record sets.

Measured on three real 100,000 record sets with no field declared, milliseconds per query, minimum of several passes:

| Records matched | Path | sift, 128d | glove, 100d | dbpedia, 1536d |
| --- | --- | --- | --- | --- |
| no filter | graph | 0.30 | 0.29 | 1.16 |
| 50,000 | graph | 3.2 | 3.7 | 5.0 |
| 10,000 | graph | 18.0 | 27.2 | 31.6 |
| 1,000 | exact | 33.1 | 39.4 | 38.4 |
| 100 | exact | 36.1 | 30.8 | 30.8 |
| 1 | exact | 38.3 | 31.6 | 34.7 |

**Declare the fields you filter on**, because undeclared a filtered search over 100,000 records costs tens of milliseconds where an unfiltered one costs a fraction of one, and it grows in proportion to the record count. Filtering on a field that few records carry does not reduce it, since the walk visits every record either way.

<br/>

### 🚀 Declaring the fields you filter on

`create(indexed_fields=[...])` builds a column for each field named, so a filter naming only declared fields is answered from those columns rather than by reading every record's metadata. It changes which records come back in no way, only what finding them costs.

```python
from zeusdb_vector_database import VectorDatabase

catalogue = VectorDatabase().create(
    "hnsw", dim=4, space="l2", indexed_fields=["lang", "tier"]
)
catalogue.add([
    {"id": "c1", "values": [0.1, 0.1, 0.1, 0.1],
     "metadata": {"lang": "en", "tier": "gold", "year": 2024}},
    {"id": "c2", "values": [0.2, 0.2, 0.2, 0.2],
     "metadata": {"lang": "es", "tier": "free", "year": 2023}},
])
query = [0.1, 0.1, 0.1, 0.1]

print(catalogue.indexed_fields)

# Answered from the columns, because every field the filter names is declared.
print([hit["id"] for hit in catalogue.search(vector=query, filter={"tier": "gold"})])

# The same record, found by reading metadata, because year was not declared.
print([hit["id"] for hit in catalogue.search(vector=query, filter={"year": 2023})])
```

*Output*
```
['lang', 'tier']
['c1']
['c2']
```

The same filter answered both ways, on three real 100,000 record sets, milliseconds per query, minimum of three passes over thirty queries:

| Records matched | Declared | Not declared |
| --- | --- | --- |
| 1 | 0.09 to 0.15 | 28.1 to 73.9 |
| 1,000 | 0.37 to 0.48 | 31.2 to 57.2 |
| 10,000 | 3.5 to 12.6 | 20.5 to 36.9 |
| 50,000 | 0.82 to 4.1 | 3.9 to 15.7 |

**Declare the fields you filter on and leave the rest out.** Eight declared fields over 100,000 records cost 6.69 MB, which is 6 percent of what the metadata already costs. A field carrying a distinct value on nearly every record costs 42 bytes a record instead of 4, so declare a document id only if you filter on it.

**A filter naming an undeclared field returns the same records**, finds them by reading metadata rather than from a column, and logs one warning naming the field. `index.indexed_fields` reads the declaration back and is empty on an index created without it.

<br/>

### 💡 Practical Filter Examples

The examples below all run against this index:

```python
from zeusdb_vector_database import VectorDatabase

vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, space="l2")
index.add([
    {"id": "doc_1", "values": [0.1, 0.1, 0.1, 0.1], "metadata": {
        "author": "Alice", "rating": 4.5, "year": 2024, "price": 29.99,
        "published": True, "tags": ["ai", "science"], "title": "The Guide",
        "filename": "report.pdf", "lang": "en"}},
    {"id": "doc_2", "values": [0.2, 0.2, 0.2, 0.2], "metadata": {
        "author": "Bob", "rating": 3.0, "year": 2023, "price": 45.00,
        "published": False, "tags": ["cooking"], "title": "A Book",
        "filename": "notes.txt", "lang": "es"}},
    {"id": "doc_3", "values": [0.3, 0.3, 0.3, 0.3], "metadata": {
        "author": "Charlie", "rating": 5.0, "year": 2026, "price": 25.00,
        "published": True, "tags": ["ai"], "title": "Theory",
        "filename": "paper.pdf", "lang": "fr"}},
])
query_embedding = [0.1, 0.1, 0.1, 0.1]
```

#### ✔️ The filter chooses what is ranked, so `top_k` is just the page size

```python
def matched(filter, top_k=10):
    return [hit["id"] for hit in index.search(vector=query_embedding, filter=filter, top_k=top_k)]

# doc_3 is the furthest of the three from the query, and it is still the only
# thing the filter admits, so it is what a page of one holds
print(matched({"author": "Charlie"}, top_k=1))
print(matched({"author": "Charlie"}, top_k=10))
```

*Output*
```
['doc_3']
['doc_3']
```

A filter matching fewer records than `top_k` returns that many results, and one matching none returns an empty list. Neither is a truncation.

#### ✔️ Common filters

```python
# Find high-quality recent documents
print(matched({"published": True, "rating": {"gte": 4.0}, "year": {"gte": 2024}}))

# Find documents by specific authors
print(matched({"author": {"in": ["Alice", "Bob"]}}))

# Find AI-related content
print(matched({"tags": {"contains": "ai"}}))

# Find documents in a price range
print(matched({"price": {"gte": 20.0, "lte": 40.0}}))

# Find documents with a specific file type
print(matched({"filename": {"endswith": ".pdf"}}))

# Match on a title prefix
print(matched({"title": {"startswith": "The"}}))

# Exclude an author
print(matched({"author": {"ne": "Alice"}}))

# Match a whole array
print(matched({"tags": ["ai"]}))

# Either a top rating or a recent year, which needs a disjunction
print(matched({"$or": [{"rating": {"gte": 5.0}}, {"year": {"gte": 2024}}]}))

# Published, and either English or cheap
print(matched({"published": True,
               "$or": [{"lang": "en"}, {"price": {"lt": 26.0}}]}))

# Everything except Bob's, including any record with no author at all
print(matched({"$not": {"author": "Bob"}}))
```

*Output*
```
['doc_1', 'doc_3']
['doc_1', 'doc_2']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_2', 'doc_3']
['doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
```

<br />

## 📝 Logging

ZeusDB Vector Database includes structured logging that works automatically out of the box while providing customization for advanced users.

### 🚀 Basic Usage - it just works!

**For most users, logging works automatically with sensible defaults:**

<!-- zeusdb:skip -->
```python
from zeusdb_vector_database import VectorDatabase
# Logging is automatically configured, no setup required

vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536)

# Operations are automatically logged with structured data
result = index.add({"ids": ids, "embeddings": vectors})
results = index.search(query_vector, top_k=5)
```

**What you get automatically:**
- ✅ **Quiet by default**, only warnings and errors outside development
- ✅ **Environment detection**, appropriate defaults for dev, prod, testing, CI and notebooks
- ✅ **Structured JSON logs** in production environments
- ✅ **Human-readable logs** in development environments
- ✅ **Operation timing** on index creation, additions, searches and saves
- ✅ **Cross-platform compatibility**

`save()` and `load()` write their progress here too, at `debug`, so they print nothing on stdout. They used to print it directly and it was not affected by any of the settings below.

### ⚙️ Intermediate Usage (Environment Variables)

**Control logging behavior with environment variables:**

#### Quick Development Debugging
```bash
export ZEUSDB_LOG_LEVEL=debug
python your_app.py
```

#### Production JSON Logging
```bash
export ZEUSDB_LOG_LEVEL=error
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=/var/log/zeusdb/app.log
python your_app.py
```

#### Environment Variables Reference

| Variable | Options | Default | Description |
|----------|---------|---------|-------------|
| `ZEUSDB_LOG_LEVEL` | `trace`, `debug`, `info`, `warn`, `error` | `warn` (dev), `error` (prod) | Controls log verbosity. `warning` and `warn` are the same level, as are `critical`, `fatal` and `error`. An unrecognised name falls back to the default. |
| `ZEUSDB_LOG_FORMAT` | `human`, `json` | `human` (dev), `json` (prod) | Output format |
| `ZEUSDB_LOG_TARGET` | `stdout`, `stderr`, `file` | `stderr` | Where logs go |
| `ZEUSDB_LOG_FILE` | `/path/to/file.log` | `zeusdb.log` | Log file path, written exactly as given (if target=file) |
| `ZEUSDB_LOG_ROTATION` | `daily`, `never` | `never` | With `daily`, a UTC date is appended to the file name |
| `ZEUSDB_LOG_CONSOLE` | `true`, `false` | Auto-detected | Force console output |
| `ZEUSDB_DISABLE_AUTO_LOGGING` | `true`, `1`, `yes` | unset | Skip automatic configuration entirely |
| `RUST_LOG` | standard `env_logger` syntax | unset | Overrides `ZEUSDB_LOG_LEVEL` for the Rust layer |

**⚠️ `warning` and `critical` are not accepted level names.** The Python layer accepts them, but the Rust layer rejects them and prints `ignoring 'zeusdb_vector_database=warning': invalid filter directive`. The bare `warn` is the opposite, accepted by Rust and rejected by Python. Use `trace`, `debug`, `info` or `error`, which both layers accept.

Under `ZEUSDB_LOG_ROTATION=daily` with `ZEUSDB_LOG_FILE=logs/app.log`, two files appear: `logs/app.log` and a dated `logs/app.log.2026-08-05`. Rotation applies to the Rust layer, which writes the dated one.

#### Smart Environment Detection
The system detects your environment and applies appropriate defaults:

- **🏭 Production** (`ENVIRONMENT=production`, or Kubernetes or Docker markers): ERROR level, JSON format, file output
- **💻 Development** (default): WARNING level, human format, console output
- **🧪 Testing** (`ENVIRONMENT=testing`, `PYTEST_CURRENT_TEST`, or `pytest` imported): CRITICAL level, minimal output
- **📓 Jupyter** (`JUPYTER_SERVER_ROOT`, `JPY_PARENT_PID`, or IPython imported): INFO level, human format
- **🔄 CI/CD** (`CI`, `GITHUB_ACTIONS`, `GITLAB_CI`): WARNING level, human format for readability

Environment variables always override the detected defaults.

### 🔧 Advanced Usage (Programmatic Control)

**For enterprise environments with existing logging infrastructure:**

#### Option 1: Disable Auto-Configuration
<!-- zeusdb:skip -->
```python
import os
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"

# Now configure your own logging before importing ZeusDB
import logging
logging.basicConfig(level=logging.INFO, format='%(message)s')

from zeusdb_vector_database import VectorDatabase  # Will respect your existing logging setup
```

#### Option 2: Programmatic Initialization
<!-- zeusdb:skip -->
```python
import os
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"

import zeusdb_vector_database

# JSON to stdout
success = zeusdb_vector_database.init_logging(level="info")

# OR JSON to a directory of daily rotating files. Pick one, not both.
# success = zeusdb_vector_database.init_file_logging(
#     log_dir="/var/log/myapp",
#     level="debug",
#     file_prefix="zeusdb"
# )

print("initialized:", success)

vdb = zeusdb_vector_database.VectorDatabase()
```

**Only the first initializer to run takes effect.** Both functions return `True` if they installed the subscriber and `False` if one was already installed, so calling both leaves the second with no effect and a `False` return. `zeusdb_vector_database.is_logging_initialized()` reports whether either has run.

**The file target is drained at exit.** Records reach the file through a background writer, so a record emitted immediately before the process ends is still in flight when it ends. Importing the package registers the drain with `atexit`, which covers a normal exit and needs no call. `zeusdb_vector_database.shutdown_logging()` runs the same drain on demand and returns `True` if it drained a file appender, or `False` if there was nothing to drain, which is the answer for the `stdout` and `stderr` targets and for a second call. It closes the file, so records emitted after it are discarded. Nothing runs on `os._exit` or on a crash, and records still in flight at either are lost.

#### Option 3: Custom Logger Integration
<!-- zeusdb:skip -->
```python
import logging
import os

# Disable auto-configuration
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"

# Set up your own logger first
logger = logging.getLogger("myapp.zeusdb")
logger.setLevel(logging.INFO)

# Configure Rust logging to match
os.environ["ZEUSDB_LOG_LEVEL"] = "info"
os.environ["ZEUSDB_LOG_FORMAT"] = "json"

from zeusdb_vector_database import VectorDatabase
# ZeusDB will integrate with your logging setup
```

### 📊 Log Output Examples

#### Human-Readable (Development)
```
2026-08-05T12:19:39.261318Z  INFO build: HNSW index created successfully operation="index_creation_complete" dim=8 space=cosine m=16 ef_construction=200 expected_size=10000 has_quantization=false duration_ms=0
2026-08-05T12:19:39.3491294Z  INFO add: Vector addition completed operation="add_vectors_complete" total_inserted=2 total_errors=0 success_rate=100.0 duration_ms=87 overwrite_mode=true final_storage_mode="raw_only"
```

#### Structured JSON (Production)
```json
{"timestamp":"2026-08-05T12:19:39.4853862Z","level":"INFO","fields":{"message":"HNSW index created successfully","operation":"index_creation_complete","dim":8,"space":"cosine","m":16,"ef_construction":200,"expected_size":10000,"has_quantization":false,"duration_ms":"0"},"target":"zeusdb_vector_database::hnsw_index","filename":"src\\hnsw_index.rs","line_number":1068,"threadId":"ThreadId(1)"}
```

### 🔍 Monitoring and Observability

#### Key Fields to Monitor
- **`operation`**: the operation name, for example `index_creation_complete`, `add_vectors_complete`, `search_complete`, `pq_training_complete`, `save_complete`, `compact_complete`
- **`duration_ms`**: timing on index creation, additions, searches, saves and compaction
- **`total_inserted`**, **`total_errors`**, **`success_rate`**: outcome of each `add()`
- **`final_storage_mode`**: whether an index is serving raw or quantized results
- **`results_count`**: results returned by a search

#### Production Alerting Examples
```bash
# Monitor error rates
grep '"level":"ERROR"' /var/log/zeusdb/app.log | wc -l

# Track search latency
grep '"operation":"search_complete"' /var/log/zeusdb/app.log | jq '.fields.duration_ms'

# Watch quantization training
grep '"operation":"pq_training' /var/log/zeusdb/app.log
```

### 🛠️ Troubleshooting

#### Common Issues

**Logs not appearing?**
```bash
# Check if auto-logging is disabled
echo $ZEUSDB_DISABLE_AUTO_LOGGING

# Verify the level is one both layers accept
ZEUSDB_LOG_LEVEL=debug python -c "import zeusdb_vector_database as z; print(z.is_logging_initialized())"
```

**File logging not working?**
```bash
# Check permissions
ls -la /path/to/log/directory

# Test with console first
ZEUSDB_LOG_TARGET=stderr ZEUSDB_LOG_LEVEL=info python your_app.py
```

A process that ends through `os._exit` or a crash skips the exit drain, so its final records never reach the file. Call `zeusdb_vector_database.shutdown_logging()` before such an exit.

**Want to see Rust logs specifically?**
```bash
# Enable trace level to see all Rust operations
ZEUSDB_LOG_LEVEL=trace python your_app.py
```

#### Performance Notes
- File logging is non-blocking: records are handed to a background writer rather than written on the calling thread. The exit drain waits for that writer to finish, for up to about a second.
- `trace` and `debug` are verbose enough to dominate runtime on a hot loop. Leave production at `error`.

### 🎯 Best Practices

#### Development
```bash
export ZEUSDB_LOG_LEVEL=debug
export ZEUSDB_LOG_FORMAT=human
```

#### Staging
```bash
export ZEUSDB_LOG_LEVEL=info
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=logs/zeusdb-staging.log
export ZEUSDB_LOG_ROTATION=daily
```

#### Production
```bash
export ENVIRONMENT=production
export ZEUSDB_LOG_LEVEL=error
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=/var/log/zeusdb/production.log
export ZEUSDB_LOG_ROTATION=daily
```

<br/>

## 📄 License

This project is licensed under the Apache License 2.0.

