Metadata-Version: 2.4
Name: waveflowdb_client
Version: 1.0.6
Summary: VectorLake SDK — Deterministic backend engine powering agent workflows
Author-email: "agentanalytics.ai" <nitin@agentanalytics.ai>
License: MIT License
        
        Copyright (c) 2026 agentanalytics.ai
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://agentanalytics.ai
Project-URL: Documentation, https://www.agentanalytics.ai/docs/waveflow-db
Keywords: vector db,VECTOR QUERY LANGUAGE,waveflowdb,agentanalytics,VQL
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: numpy
Requires-Dist: tqdm
Dynamic: license-file

# WaveflowDB SDK — VectorLake Python Client

A Python SDK for interacting with **WaveflowDB** and performing **VQL (Vector Query Language)** brace-based semantic retrieval.

This SDK provides:
- Full `Config` management via constructor args, environment variables, or `.env` file
- A Pinecone/Chroma-style high-level API (`VectorLake` / `Index`) alongside the full low-level `VectorLakeClient`
- Document ingestion — direct payload mode or batched filesystem mode
- Document deletion — remove documents from the index by filename
- Targeted insertion — insert documents at a controlled position in the corpus
- Intelligent document sync with MD5-based change detection
- Resumable batch uploads with checkpoint-based recovery (`RESUME=True`)
- Semantic search and top-matching-doc retrieval with hybrid filtering
- Namespace and document metadata queries
- **Custom per-file metadata** — attach your own fields via `files_meta` / `metadatas`, keyed by filename
- **Automatic filename sanitization** — messy source filenames are cleaned on disk before chunking
- Structured CSV + JSONL logging for performance, errors, and failed files
- Simple `{stem}_part{num}.txt` chunk naming for idempotent re-runs
- **Image & OCR extraction** — local (PyMuPDF + Tesseract) and cloud engines (AWS Textract, GCP Vision, Azure Doc Intelligence, Mathpix, LlamaParse)
- **Scale-out cluster** — `cluster_controller.py` + `cluster_api.py` for multi-node deployments

> **Note:** Parallel and sync processing modes have been removed. Sequential is the default and recommended mode. Streaming suits large OCR-heavy pipelines.

---

## Quick Start

```bash
pip install waveflowdb_client python-dotenv
cp .env.example .env    # fill in your credentials
# drop source files into ./upload
python run.py           # ACTION = "add" by default
```

Or, using the high-level API directly in code:

```python
from vectorlake import VectorLake

vl = VectorLake(api_key="YOUR_API_KEY", user_id="you@company.com")

index = vl.Index("support_docs")          # namespace — created on first add()
index.add(files=["faq.pdf", "policy.docx"])

results = index.query("what's the refund policy?", top_k=5)
index.delete(ids=["faq.pdf"])
```

---

## Installation

```bash
pip install waveflowdb_client
```

Optional extras for richer file support:

```bash
pip install PyPDF2 python-docx tqdm
```

Optional extras for local OCR:

```bash
pip install pymupdf pytesseract    # requires Tesseract binary on PATH
```

Optional extras for cloud OCR (install only the engine you use):

```bash
pip install boto3                           # AWS Textract
pip install google-cloud-vision             # GCP Vision
pip install azure-ai-documentintelligence  # Azure Document Intelligence
pip install requests                        # Mathpix + LlamaParse (REST APIs)
```

---

## Configuration

All settings are read from the environment (or `.env`). Constructor arguments take priority.

**`.env` file:**

```
VECTOR_LAKE_API_KEY=your_api_key_here
VECTOR_LAKE_HOST=https://waveflow-analytics.com
USER_ID=your@email.com
NAMESPACE=your_namespace
```

**Constructor arguments:**

```python
from waveflowdb_client import Config, VectorLakeClient

cfg = Config(
    api_key="your_api_key_here",
    host="https://waveflow-analytics.com",
    vector_lake_path="/path/to/documents",
)
client = VectorLakeClient(cfg)
```

### Core environment variables

| Env var | Default | Description |
|---|---|---|
| `VECTOR_LAKE_API_KEY` | required | API key |
| `USER_ID` | required | Your user identifier |
| `NAMESPACE` | required | Target namespace / lake name |
| `VECTOR_LAKE_HOST` | `https://waveflow-analytics.com` | Server base URL |
| `VECTOR_LAKE_PATH` | `upload` | Directory containing source files |
| `VECTOR_LAKE_LOG_DIR` | `logs` | Directory for log files |
| `VECTOR_LAKE_MAX_FILES_PER_BATCH` | `100` | Max chunk files per upload batch |
| `VECTOR_LAKE_MAX_BATCH_SIZE_MB` | `1` | Max total batch size in MB |
| `VECTOR_LAKE_TIMEOUT` | `240` | HTTP request timeout in seconds |
| `VECTOR_LAKE_MAX_RETRIES` | `2` | Retry attempts on transient errors |
| `VECTOR_LAKE_ENABLE_OCR` | `true` | Enable local Tesseract OCR |
| `VECTOR_LAKE_OCR_LANG` | `eng` | Tesseract language code |
| `VECTOR_LAKE_STREAMING_MINIBATCH_FILES` | `10` | Chunks per streaming minibatch |
| `VECTOR_LAKE_STREAMING_MINIBATCH_SIZE_MB` | `2` | MB per streaming minibatch |

---

## High-Level API — `VectorLake` / `Index`

For most use cases you don't need the low-level client at all. `VectorLake` / `Index` is a thin, Pinecone/Chroma-shaped wrapper over `VectorLakeClient` — same retry, batching, chunking, sanitization, and OCR behaviour underneath.

```python
from vectorlake import VectorLake

vl = VectorLake(api_key="YOUR_API_KEY", user_id="you@company.com")
index = vl.Index("support_docs")

index.add(files=["faq.pdf", "policy.docx"])
index.add()                                     # every supported file in vector_lake_path
index.add(documents=["full text..."], ids=["a.txt"])   # in-memory, Chroma-style

results = index.query("refund policy", top_k=5)
index.update(files=["policy.docx"])             # re-index a changed file
index.delete(ids=["faq.pdf"])

index.get()                                     # list documents in this index
index.describe_index_stats()                    # namespace metadata
vl.list_indexes()                                # every namespace you can see
vl.health()                                      # is the server up?
```

`upsert()` is an alias for `add()`.

### Custom metadata via `metadatas`

`index.add()` / `index.update()` accept a `metadatas` argument, forwarded to the underlying client's `files_meta`. It accepts two shapes:

- **dict, keyed by original filename** — the recommended shape when using `files=` (path mode). Resolved per source file regardless of chunk-splitting or batch-interleaving order:

  ```python
  index.add(
      files=["report.pdf", "notes.txt"],
      metadatas={
          "report.pdf": {"department": "finance", "year": 2026},
          "notes.txt":  {"author": "alice"},
      },
  )
  ```

  Files not mentioned as a key just get the default `{"filename", "extension", "timestamp"}` stub.

- **list, positionally aligned with `documents`/`ids`** — the original in-memory-mode behaviour:

  ```python
  index.add(
      documents=["full text one", "full text two"],
      ids=["a.txt", "b.txt"],
      metadatas=[{"author": "alice"}, {"author": "bob"}],
  )
  ```

  A list must be the same length as `documents`, and is not accepted together with `files=` — path mode requires the dict shape, since there's no fixed per-document ordering to align a list against. Passing a list with `files=` raises a `ValueError` explaining this rather than silently dropping your metadata.

See [`files_meta` — Custom Metadata](#files_meta--custom-metadata) below for the equivalent behaviour on the low-level `VectorLakeClient`, and [Filename Sanitization](#filename-sanitization) for why the dict is keyed by filename rather than position.

---

## Processing Modes

Two upload modes are supported. Set `MODE` at the top of `run.py`.

### `sequential` (default)

All source files are chunked to disk first, then batches are uploaded one at a time.

- **Resume:** if a run is interrupted, set `RESUME = True` (default). The SDK reads the checkpoint and picks up from the next unprocessed batch.
- **Re-chunking guard:** each successfully uploaded source file gets an MD5 sidecar. On the next run, if the file is unchanged, chunking and OCR are skipped entirely. If changed, stale chunks are purged and the file is re-extracted automatically.
- **Manual resume:** set `START_FROM_BATCH = N` to skip the first N−1 batches (ignored when `RESUME = True` and a checkpoint exists).

### `streaming`

Extraction and upload are pipelined: the SDK begins uploading the first minibatch as soon as it is ready, while OCR of later files proceeds concurrently. Best for large collections with expensive PDF/image extraction.

- **Resume:** completed file stems are stored in the checkpoint. On restart, already-uploaded files are skipped automatically.
- `start_from_batch` / `end_batch` are not supported in streaming mode.

---

## Checkpointing

Checkpoints are written to:

```
{VECTOR_LAKE_PATH}/chunks/checkpoint_{USER_ID}_{NAMESPACE}_{operation}.json
```

The filename encodes **user ID**, **namespace**, and **operation type** (`add_docs`, `refresh_docs`, `insert_docs`) so checkpoints for different namespaces or operations are always independent and never collide.

**Sequential checkpoint:**

```json
{
  "last_ok_batch": 14,
  "operation": "add_docs",
  "user_id": "alice_at_example.com",
  "namespace": "finance_q3",
  "mode": "sequential",
  "updated_at": "2025-08-01T10:22:00+00:00"
}
```

**Streaming checkpoint:**

```json
{
  "completed_stems": ["report_2024", "appendix_a"],
  "operation": "add_docs",
  "user_id": "alice_at_example.com",
  "namespace": "finance_q3",
  "mode": "streaming",
  "updated_at": "2025-08-01T10:22:00+00:00"
}
```

Checkpoints are **cleared automatically** when a run completes with zero failures. To force a fresh start, set `RESUME = False` or delete the checkpoint file.

---

## Failure Auditing

Every failed batch is appended to `logs/failed_files.jsonl`:

```json
{
  "timestamp": "2025-08-01T10:23:45Z",
  "operation": "add_docs",
  "user_id": "alice_at_example.com",
  "namespace": "finance_q3",
  "batch_num": 7,
  "files": ["report_part3.txt", "report_part4.txt"],
  "error_type": "HTTP_500",
  "error_message": "Internal server error",
  "traceback": null,
  "extra": { "elapsed_ms": 312.4, "processing_time": 0.31 }
}
```

Failures are written both from `run.py` and from within `client.py` directly, so every failure path is covered regardless of how the SDK is invoked.

---

## `run.py` — Operations Launcher

Set `ACTION` at the top of `run.py`, then run `python run.py`.

```
  run.py
  │
  ├── UPLOAD
  │   ├── "add"     — reads all files from VECTOR_LAKE_PATH as new documents
  │   ├── "refresh" — re-chunks and re-uploads existing documents
  │   ├── "update"  — alias for "refresh"
  │   ├── "insert"  — insert documents at a controlled position
  │   └── "delete"  — remove documents by name (requires FILES_TO_DELETE)
  │
  ├── QUERY
  │   └── "query"   — semantic search (flat or flat_filter)
  │
  └── INFO
      └── "health"  — ping server
      └── "info"    — list indexed documents
```

All upload operations support both **path mode** (reads from `VECTOR_LAKE_PATH`) and **direct mode** (supply `files_name` + `files_data` inline), except `delete` which only needs filenames.

### Key settings in `run.py`

```python
ACTION           = "add"          # operation to run
MODE             = "sequential"   # "sequential" | "streaming"
RESUME           = True           # auto-resume from checkpoint
START_FROM_BATCH = 1              # ignored when RESUME=True and checkpoint exists
END_BATCH        = None           # cap the run; None = run all batches
FILES_TO_DELETE  = []             # required for ACTION="delete"
```

---

## Supported File Types

| Extension | Processing strategy |
|---|---|
| `txt`, `py` | Plain text, paragraph-chunked |
| `ipynb` | Code cells extracted, paragraph-chunked |
| `pdf` | Text extracted via PyMuPDF; embedded images OCR'd if enabled |
| `docx` | Text + inline images extracted; images OCR'd if enabled |
| `csv` | Row-safe split; header preserved in every chunk |
| `xlsx` | Multi-section workbook flattened to CSV rows; row-safe split (standard CSV, proper quoting), header preserved in every chunk |
| `jsonl`, `ndjson` | Record-safe split; each line validated as JSON |
| `json` | Kept atomic — never split |
| `png`, `jpg`, `jpeg`, `tiff`, `tif`, `bmp`, `webp` | Full-image OCR via local Tesseract or cloud engine |

---

## Chunk Naming

All chunks follow this convention:

```
{stem}_part{num}.txt
```

Example — a PDF split into 3 parts:
```
annual_report_part1.txt
annual_report_part2.txt
annual_report_part3.txt
```

Chunks already on disk are **reused on re-run** — the pipeline is idempotent and safe to restart after a crash.

### Filename sanitization

Before any file is chunked, the SDK checks the filename against the same rules used by `clean_filename()`: illegal filesystem characters, runs of whitespace, and Windows-reserved names are normalized away. If a source file's name isn't already clean, **the file is renamed on disk, in place**, to its cleaned equivalent — e.g. `"Q3 Report (final).pdf"` → `"Q3_Report_final.pdf"` — before extraction ever runs.

This applies automatically in both `sequential` and `streaming` path-mode uploads (`files=` or auto-discovery), and to the file lists `run.py`/`run_add()` build for MD5 change-detection and streaming recovery. You don't need to do anything — but be aware that:

- Source filenames in `VECTOR_LAKE_PATH` **will change** the first time a messy-named file is processed.
- Chunk filenames, `.md5` sidecars, checkpoint entries, and `files_meta.filename` values are always derived from the *clean* name, so everything downstream stays consistent.
- If a cleaned name would collide with a different, already-existing file, the rename is skipped and a warning is logged — nothing is overwritten.
- The rename is idempotent: re-running against an already-sanitized directory does nothing.

---

## Image & OCR Extraction

OCR-derived text is wrapped in provenance markers inside chunk files:

```
<<<IMAGE_EXTRACT source="scan.pdf" page=2 image_index=1 method="aws_textract">>>
... extracted text ...
<<<END_IMAGE_EXTRACT>>>
```

Use the helper functions to work with these markers:

```python
from waveflowdb_client import strip_image_markers, list_image_regions

clean_text = strip_image_markers(chunk_text)   # markers removed, OCR text kept
regions    = list_image_regions(chunk_text)    # list of {source, page, image_index, method, ocr_text}
```

### Local OCR (Tesseract)

Enabled by default when `pymupdf` and `pytesseract` are installed.

```python
cfg = Config(api_key="...", enable_ocr=True, ocr_lang="eng")
```

### Cloud OCR Providers

| `cloud_ocr_provider` | Provider | Best for |
|---|---|---|
| `"aws"` | AWS Textract | Tables, forms, handwriting |
| `"gcp"` | Google Cloud Vision | Highest accuracy, 100+ languages |
| `"azure"` | Azure Document Intelligence | Forms, invoices, custom models |
| `"mathpix"` | Mathpix | Math, scientific papers, LaTeX |
| `"llama_parse"` | LlamaParse | Complex layouts, MDX |

Set `cloud_ocr_strategy="hybrid"` (default) to use local OCR first with cloud fallback. Set `"cloud"` to always use the cloud provider.

```python
# AWS Textract — hybrid
cfg = Config(api_key="...", cloud_ocr_provider="aws", aws_region="eu-west-1")

# LlamaParse — cloud only
cfg = Config(
    api_key="...",
    cloud_ocr_provider="llama_parse",
    cloud_ocr_strategy="cloud",
    llama_parse_api_key="...",
)
```

---

## Client API Reference

All public methods return a plain `dict` and **never raise** — errors surface as `{"success": False, "error": "...", "message": "..."}`.

### `add_documents`

Uploads **new** documents to the index.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `user_id` | str | required | User identifier |
| `vector_lake_description` | str | required | Target namespace |
| `files_name` + `files_data` | List[str] | — | **Direct mode**: names and string contents (equal length) |
| `files` | List[str] | `None` | **Path mode**: filenames to read from `vector_lake_path` |
| `start_from_batch` | int | `1` | Resume point for path mode |
| `end_batch` | int | `None` | Upper batch bound (inclusive) |
| `processing_mode` | str | `"sequential"` | `"sequential"` or `"streaming"` |
| `intelligent_segmentation` | bool | `True` | Server-side segmentation |
| `files_meta` | dict \| List[dict] | `None` | Per-file metadata. **Dict, keyed by original filename** (recommended for path mode — resolved per source file regardless of chunk order) or a positional **list** (legacy — reliable only in direct mode). Auto-generated `{filename, extension, timestamp}` stub used when omitted or unmatched. See [`files_meta` — Custom Metadata](#files_meta--custom-metadata). |
| `session_id` | str | `None` | Optional session token |

### `refresh_documents`

Updates **existing** documents. Stale chunks are purged and regenerated before uploading. Accepts the same parameters as `add_documents`, including the dict/list `files_meta` shapes above.

### `delete_documents`

Removes documents by filename. Lightweight metadata call — no chunking required.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `user_id` | str | required | User identifier |
| `vector_lake_description` | str | required | Target namespace |
| `files_name` | List[str] | required | Basenames to delete (at least one) |
| `session_id` | str | `None` | Optional session token |

### `insert_documents`

Inserts documents at a controlled position in the corpus. Supports both path mode and direct mode, matching the calling convention of `add_documents` and `refresh_documents` exactly.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `user_id` | str | required | User identifier |
| `vector_lake_description` | str | required | Target namespace |
| `files_name` + `files_data` | List[str] | — | **Direct mode**: names and string contents |
| `files` | List[str] | `None` | **Path mode**: filenames from `vector_lake_path` |
| `start_from_batch` | int | `1` | Resume point for path mode |
| `end_batch` | int | `None` | Upper batch bound |
| `processing_mode` | str | `"sequential"` | `"sequential"` or `"streaming"` |
| `intelligent_segmentation` | bool | `True` | Server-side segmentation |
| `files_meta` | dict \| List[dict] | `None` | Per-file metadata — same dict/list shapes as `add_documents` |
| `session_id` | str | `None` | Optional session token |

### `get_matching_docs`

Retrieves top-matching document chunks using semantic search with optional hybrid filtering.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `query` | str | required | Natural language or VQL search query |
| `user_id` | str | required | User identifier |
| `vector_lake_description` | str | required | Target namespace |
| `pattern` | str | `"static"` | `"static"` for indexed docs; `"dynamic"` for inline files |
| `hybrid_filter` | bool | `False` | Enable keyword hybrid filter. **Required when `search_type` is set** |
| `search_type` | str | `None` | `"flat"` — full-corpus fusion. `"flat_filter"` — semantic over filtered candidates only |
| `top_docs` | int | `10` | Maximum chunks to return |
| `threshold` | float | `0.2` | Similarity score cutoff |
| `with_data` | bool | `False` | Include raw chunk text in response |
| `files_name` + `files_data` | List[str] | `None` | Dynamic mode: temporary files to search |
| `session_id` | str | `None` | Optional session token |

### `health_check`

Pings the server and confirms namespace connectivity.

### `get_namespace_details`

Returns storage and quota metadata for one or all namespaces belonging to the user.

### `get_docs_information`

Returns document-level metadata, optionally filtered by keyword (`keyword`, `threshold=70`).

### `full_corpus_search`

Full-text keyword search across all documents. Complements semantic search for exact-match lookups.

---

## Search Modes

### `search_type="flat"` — Full Corpus Fusion

Semantic search and hybrid filter run independently across **all** documents, then fused into a single ranked list.

```
  Tier 1: matched BOTH filter and semantic  →  highest score
  Tier 2: filter match only
  Tier 3: semantic match only (fallback)
```

Best for: exploratory search, maximum recall.

### `search_type="flat_filter"` — Filtered Semantic Search

Hybrid filter runs first as a **hard gate** — only documents that pass are eligible for semantic ranking.

Best for: targeted retrieval when filter criteria are strong.

---

## VQL — Vector Query Language

Use braces `{…}` to pin your search to documents or passages that contain specific terms.

```python
QUERY = """select top 5
where query is "patients with high sodium levels"
contains {sodium} {(clinical trial)} {PID 555}"""
```

### Syntax

```
{A B}             implicit AND: both A and B must match
{A or B}          either A or B
{(A B) or C}      phrase "A B", or term C
{A} {B}           groups combine with implicit AND
```

### Structured data filtering (JSON, NDJSON, CSV)

Because the tokeniser splits on `:`, `,`, `|`, `=`, you can filter by exact field:value pairs:

```python
# Log file: find a specific transaction
QUERY = """select top 1 where query is "event_type"
contains {transaction_id:c00d7eac-5b86-439e-956b-118583df6d67}"""

# CSV: filter by column values
QUERY = """select top 10 where query is "shipped orders from Delhi"
contains {status:shipped} {city:Delhi}"""
```

### Important rules

- Short tokens (≤ 2 chars) and pure numbers are not indexed — use `{context_status_code:400}` not `{400}`
- Token must appear exactly as written in the source
- If an entire brace group normalises to empty, it contributes no constraint (not zero results)
- If no filter hits, the system falls back to full semantic search automatically

---

## `files_meta` — Root File Type Identification

Every batch upload automatically attaches a `files_meta` array. Each entry carries at minimum:

```json
{ "filename": "invoice_part1.txt", "extension": "pdf", "timestamp": "2026-08-05T14:22:07+00:00" }
```

The `extension` field tells the backend the **original source file type** before chunking. This is critical for format-aware indexing — CSV triggers row-level tokenisation, PDF/DOCX use paragraph tokenisation, etc. The SDK resolves this automatically by looking up the source file alongside the chunk on disk.

The `timestamp` field is a UTC ISO-8601 string (`YYYY-MM-DDTHH:MM:SS+00:00`) stamped automatically for every file — no configuration needed. All files processed together in the same batch/upload call share one timestamp value, reflecting when that batch was sent. If you supply your own `"timestamp"` key via custom metadata (see below), your value takes precedence over the auto-generated one.

### `files_meta` — Custom Metadata

Beyond `filename`/`extension`, you can attach **your own fields** — department, author, tags, whatever your application needs — via the `files_meta` parameter on `add_documents`, `refresh_documents`, `run_insert_path`, and `run_insert_path_direct` (or `metadatas` on the high-level `Index.add()` / `Index.update()`).

Two shapes are accepted:

**Dict, keyed by original filename** (recommended for path mode):

```python
client.add_documents(
    user_id="u1",
    vector_lake_description="my_lake",
    files=["report.pdf", "notes.txt"],
    files_meta={
        "report.pdf": {"department": "finance", "year": 2026},
        "notes.txt":  {"author": "alice"},
    },
)
```

Each chunk's *real source file* is resolved automatically — regardless of how the SDK happens to split it into multiple parts or interleave it with other files across batches — and your dict entry is merged on top of the auto-generated `{"filename", "extension", "timestamp"}` stub. Files you don't mention just get the default stub. Because lookups key off the filename (after [sanitization](#filename-sanitization), if any), this shape is order-independent and safe under any batching strategy, including `streaming` mode.

**List, positionally aligned with files/chunks** (legacy):

```python
client.run_insert_path_direct(
    user_id="u1",
    vector_lake_description="my_lake",
    files_name=["appendix.txt"],
    files_data=["Section A content..."],
    files_meta=[{"filename": "appendix.txt", "extension": "txt", "author": "Bob"}],
)
```

Reliable in **direct mode**, where you control the exact `files_name`/`files_data` order yourself. Fragile in path mode, where the SDK controls chunk-splitting and batch interleaving — prefer the dict shape there.

`None` (the default) sends only the auto-generated `{"filename", "extension", "timestamp"}` stub for every file.

---

## Log Files

| File | Contents |
|---|---|
| `logs/run_operations.log` | Full structured log of every run (DEBUG+) |
| `logs/failed_files.jsonl` | Append-only audit trail of every failed batch |
| `logs/api_errors.csv` | Per-request API errors with batch correlation |
| `logs/performance.csv` | Per-request latency, payload size, HTTP status |
| `logs/skipped_files.csv` | Files skipped due to unsupported extension etc. |
| `{VECTOR_LAKE_PATH}/chunks/run_<id>.json` | Per-run batch outcome report |

---

## Error Handling

All exceptions inherit from `VectorLakeError` and expose `.to_response()`.

| Exception | When raised |
|---|---|
| `ConfigError` | API key missing or config invalid at init |
| `ValidationError` | Mismatched `files_name`/`files_data` lengths, mismatched `metadatas`/`documents` lengths, or other pre-flight failures |
| `InvalidSearchTypeError` | `search_type` not `"flat"` or `"flat_filter"` |
| `DocumentNotFoundError` | Filename cannot be resolved in the index |
| `UnsupportedFileTypeError` | File extension not in `allowed_extensions` |
| `FileProcessingError` | File I/O, encoding, or parsing failure |
| `APIError` | HTTP 4xx/5xx. Carries `.status_code` and `.response_text` |
| `ThrottleError` | HTTP 429 rate limit. Carries `.retry_after` |

### Retry and backoff

```
  Request attempt 1
    ├── HTTP 429     → wait Retry-After (or 2^attempt s) → retry
    ├── Timeout      → wait 2^attempt s → retry
    ├── ConnError    → wait 2^attempt s → retry
    ├── HTTP 4xx/5xx → return error dict immediately (no retry)
    └── Success      → return response dict

  Max retries: 2  (VECTOR_LAKE_MAX_RETRIES)
```

---

## Scale-Out Cluster

For production deployments, `cluster_controller.py` + `cluster_api.py` act as a transparent reverse proxy. Point your SDK's `host` at the controller — zero changes to any SDK call.

```
  App SDK
      │  POST /upload/add_docs
      ▼
  cluster_api.py  (FastAPI, port 8080)
      │
      ├── Upload → UploadRouter → single node (co-located or round-robin)
      └── Query  → QueryRouter  → fan-out to all nodes holding namespace
                                   → merge results → return to SDK
```

### Quick start

```bash
pip install fastapi uvicorn httpx

# Single node (dev)
export DEFAULT_NODE_HOST=http://localhost:9000
export DEFAULT_NODE_API_KEY=sk-dev-key
python cluster_api.py

# Multi-node — nodes.json
export NODES_FILE=nodes.json
export NS_INDEX_PATH=namespace_index.json
python cluster_api.py
```

```json
// nodes.json
[
  {"id": "vl1", "host": "https://vl1.example.com", "api_key": "sk-1", "weight": 2},
  {"id": "vl2", "host": "https://vl2.example.com", "api_key": "sk-2", "weight": 1}
]
```

```python
# SDK — only change is host
cfg = Config(api_key="your-key", host="http://controller:8080")
```

### Proxied endpoints

All SDK endpoints are proxied. Upload calls route to one node; query calls fan out to all nodes holding the namespace and merge results (sort by score, de-dup by filename).

### Admin API

| Method | Path | Description |
|---|---|---|
| `GET` | `/admin/status` | Full cluster snapshot |
| `GET` | `/admin/nodes` | List registered nodes |
| `GET` | `/admin/namespaces` | Namespace → node map |
| `POST` | `/admin/nodes/register` | Register a node at runtime |
| `DELETE` | `/admin/nodes/{node_id}` | Deregister a node |

---

## Directory Layout

```
project/
├── run.py                  # launcher — set ACTION here
├── .env                    # credentials (not committed)
├── upload/                 # source files go here
│   └── chunks/             # auto-generated: chunk files, MD5 sidecars, checkpoints, run reports
├── logs/                   # auto-generated: run_operations.log, failed_files.jsonl, CSVs
└── waveflowdb_client/      # SDK package
    ├── __init__.py
    ├── client.py
    ├── config.py
    ├── exceptions.py
    ├── extractors.py
    ├── index.py             # high-level VectorLake / Index facade
    ├── models.py
    ├── run_utils.py
    └── utils.py
```

---

## Support

For API or platform support, visit: **https://db.agentanalytics.ai**

---

## License

Copyright DIBR tech private ltd.
