Metadata-Version: 2.4
Name: waveflowdb_client
Version: 1.0.3
Summary: VectorLake SDK — Deterministic backend engine powering agent workflows
Author-email: "agentanalytics.ai" <nitin@agentanalytics.ai>
License: MIT License
        
        Copyright (c) 2026 agentanalytics.ai
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://agentanalytics.ai
Project-URL: Documentation, https://www.agentanalytics.ai/docs/waveflow-db
Keywords: vector db,VECTOR QUERY LANGUAGE,waveflowdb,agentanalytics,VQL
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: numpy
Requires-Dist: tqdm
Dynamic: license-file

# WaveflowDB SDK — VectorLake Python Client v2.0.0

A Python SDK for interacting with **WaveflowDB** and performing **WaveQL (VQL) brace-based semantic retrieval**.

This SDK provides:
* Full `Config` management via constructor args, environment variables, or `.env` file
* Document ingestion — direct payload mode or batched filesystem mode
* Intelligent document sync with MD5-based change detection
* Resumable batch uploads with `start_from_batch` / `end_batch` range control
* Semantic search and top-matching-doc retrieval with hybrid filtering
* Namespace and document metadata queries
* Structured CSV logging for performance, errors, and skipped files
* Simple `{stem}_part{num}.txt` chunk naming for idempotent re-runs

---

## 📌 Overview

Vector Lake is an **agentic backend for enterprises to build AI products**, enabling:
* Natural-language structured filtering through **WaveQL (VQL)**
* Hybrid ranking (Filter + Semantic)
* Zero-schema ingestion (no JSON schemas required)
* SQL-like logical joins on raw text
* Automatic semantic fallback when filters are absent

---

## 🚀 Getting Started

### 1. Install Dependencies

```bash
pip install waveflowdb_client
```

Optional extras for richer file support:

```bash
pip install PyPDF2 python-docx python-dotenv tqdm
```

### 2. Configure API Credentials

The SDK resolves settings in this priority order: **constructor args → environment variables → `.env` file → hard-coded defaults**.

**Option A — `.env` file**

```
VECTOR_LAKE_API_KEY=your_api_key_here
VECTOR_LAKE_HOST=https://waveflow-analytics.com
USER_ID=your@email.com
NAMESPACE=your_namespace
```

**Option B — Constructor arguments**

```python
from waveflowdb_client import Config, VectorLakeClient

cfg = Config(
    api_key="your_api_key_here",
    host="https://waveflow-analytics.com",
    vector_lake_path="/path/to/documents",
)
client = VectorLakeClient(cfg)
```

---

## 🗂️ Supported File Types

The allowed extension list is the **single source of truth** defined in `Config` (or `VECTOR_LAKE_ALLOWED_EXTENSIONS`). The default set is:

| Extension | Processing strategy |
|-----------|-------------------|
| `txt` | Plain text, paragraph-chunked |
| `py` | Plain text read, paragraph-chunked |
| `ipynb` | Code cells extracted, paragraph-chunked |
| `pdf` | Text extracted via PyPDF2, paragraph-chunked |
| `docx` | Text extracted via python-docx, paragraph-chunked |
| `csv` | Row-safe split; header preserved in every chunk |
| `jsonl` / `ndjson` | Record-safe split; each line validated as JSON |
| `json` | Kept atomic — never split |

To add or remove extensions without changing code, update `VECTOR_LAKE_ALLOWED_EXTENSIONS` in your `.env`.

---

## 📦 Chunk Naming

All chunks written to disk follow this simple convention:

```
{stem}_part{num}.txt
```

Examples — a PDF that splits into 3 parts:
```
electrolytes-test-report_part1.txt
electrolytes-test-report_part2.txt
electrolytes-test-report_part3.txt
```

All formats (pdf, docx, py, ipynb, txt, etc.) produce `.txt` chunk files. The `VersionedChunkName` model provides `build()` and `parse()` helpers and an `is_versioned()` static method. Chunks already on disk are **reused on re-run** — the pipeline is idempotent and safe to restart after a crash.

---

## 🔄 Document Sync — How Uploads Work

```
  YOUR FOLDER (vector_lake_path/)
  ┌─────────────────────────────────────────────┐
  │  report.pdf   notes.txt   data.csv  ...     │
  └─────────────────────────┬───────────────────┘
                             │
                    sync_documents()
                    classifies each file
                             │
          ┌──────────────────┼───────────────────┐
          ▼                  ▼                   ▼
     No chunk          Chunk exists,        Chunk exists,
      on disk           MD5 matches          MD5 differs
          │                  │                   │
        NEW              UNCHANGED             CHANGED
          │                  │                   │
      add_docs()           skip             refresh_docs()
          │                                      │
          └──────────────────┬───────────────────┘
                             ▼
                    Upload in batches
                    (resumable via start_from_batch)
```

`sync_documents()` is the **recommended upload method** for most use cases. Use `add_documents()` or `refresh_documents()` directly only when you are certain all files are brand-new or all already exist in the namespace.

### Resumable Batch Ranges

All upload methods support `start_from_batch` and `end_batch`. If a long run is interrupted, set `START_FROM` to the last successful batch + 1 and re-run:

```
  Batch 1 ✓  Batch 2 ✓  Batch 3 ✓  [CRASH]  Batch 4 ...  Batch 39
  ──────────────────────────────────────────────────────────────────
                                     ↑
                              start_from_batch=4
                              Re-run resumes here
```

---

## Search Modes — `flat` vs `flat_filter`

The two `search_type` values represent fundamentally different retrieval strategies. Both require `hybrid_filter=True`.

### `search_type="flat"` — Full Corpus Fusion

Both semantic search and hybrid filter run **independently across ALL documents** in the namespace. Their results are then combined by the proprietary fusion algorithm, which re-ranks everything into a single fused list.

```
  Your Query
      │
      ├─────────────────────────────────┐
      ▼                                 ▼
  Semantic Search               Hybrid Filter
  (ALL documents                (keyword scan
   in namespace)                 ALL documents)
      │                                 │
      │   ┌─────────────────────────────┘
      ▼   ▼
  Proprietary Fusion Algorithm
  (combines + re-ranks both result sets)
      │
      ▼
  ┌─────────────────────────────────────────────────┐
  │  Tier 1: matched BOTH  →  highest fused score   │
  │  Tier 2: filter match only                      │
  │  Tier 3: semantic match only (fallback)         │
  └─────────────────────────────────────────────────┘
```

**Use `flat` for:** exploratory search, maximum recall, when you want every document to have a chance to surface.

---

### `search_type="flat_filter"` — Filtered Semantic Search

The hybrid filter runs first as a **hard gate** — only documents that pass are eligible for semantic search. Semantic ranking then runs only over that filtered candidate set.

```
  Your Query
      │
      ▼
  Hybrid Filter
  (keyword scan ALL documents)
      │
      ▼
  Filtered candidate set
  (only docs that passed the filter)
      │
      ▼
  Semantic Search
  (only over filtered candidates)
      │
      ▼
  Final ranked results
  (higher precision, smaller result set)
```

**Use `flat_filter` for:** targeted retrieval, when filter criteria are strong, when you want tighter and more focused results.

---

### Quick Comparison

```
  ┌─────────────────┬────────────────────────────┬────────────────────────────┐
  │                 │           flat             │        flat_filter         │
  ├─────────────────┼────────────────────────────┼────────────────────────────┤
  │ Semantic scope  │ ALL documents              │ Filter-passing docs only   │
  │ Filter role     │ Scoring signal (boosts)    │ Hard gate (excludes)       │
  │ Result style    │ Fused, broad               │ Precise, narrower          │
  │ Best for        │ Exploratory search         │ Targeted retrieval         │
  │ hybrid_filter   │ Required (True)            │ Required (True)            │
  └─────────────────┴────────────────────────────┴────────────────────────────┘
```

---

## `run.py` — Local Admin Starter Script

`run.py` is a ready-to-use local launcher for admin and testing purposes. Set your credentials via `.env`, drop files into your upload folder, uncomment exactly one function in `__main__`, and run:

```bash
python run.py
```

### Setup

```python
# run.py — top of file
API_KEY          = os.getenv("VECTOR_LAKE_API_KEY")
HOST             = os.getenv("VECTOR_LAKE_HOST", "https://waveflow-analytics.com")
VECTOR_LAKE_PATH = os.getenv("VECTOR_LAKE_PATH", "upload")
USER_ID          = os.getenv("USER_ID")
NAMESPACE        = os.getenv("NAMESPACE")

START_FROM = 1    # ← set to (last completed batch + 1) to resume
```

### Function Map

```
  run.py
  │
  ├── UPLOAD ──────────────────────────────────────────────────────────────
  │   ├── run_add_direct()          Add new files via string content
  │   ├── run_add_path()            Add new files from disk (batched)
  │   ├── run_refresh_direct()      Update existing files via string content
  │   ├── run_refresh_path()        Update existing files from disk (batched)
  │   ├── run_sync()                Auto-classify & route (recommended)
  │   └── run_sync_dry_run()        Preview sync plan, no upload
  │
  ├── QUERY ───────────────────────────────────────────────────────────────
  │   ├── run_match_static(q)             flat search, static index
  │   ├── run_match_filtered(q)           flat_filter search, static index
  │   ├── run_match_dynamic(q)            flat search, temp docs
  │   └── run_match_dynamic_filtered(q)   flat_filter search, temp docs
  │
  └── INFO ────────────────────────────────────────────────────────────────
      ├── run_health()              Ping server
      ├── run_namespace_details()   Storage & quota info
      └── run_docs_info()           List all indexed documents
```

### Full Script

```python
"""
run.py — Simple launcher for VectorLake SDK v2.0.0

QUICK START
───────────
1. Copy .env.example → .env and fill in your credentials.
2. Drop your files into VECTOR_LAKE_PATH (default: ./upload).
3. Uncomment exactly ONE action at the bottom and run:  python run.py

RESUMING A BATCH UPLOAD
───────────────────────
Set START_FROM to (last successful batch + 1) and re-run.
"""

import json
import os
from dotenv import load_dotenv

load_dotenv()

from waveflowdb_client import Config, VectorLakeClient

# ── Configuration ─────────────────────────────────────────────────────────────
API_KEY          = os.getenv("VECTOR_LAKE_API_KEY")
HOST             = os.getenv("VECTOR_LAKE_HOST", "https://waveflow-analytics.com")
VECTOR_LAKE_PATH = os.getenv("VECTOR_LAKE_PATH", "upload")
USER_ID          = os.getenv("USER_ID")
NAMESPACE        = os.getenv("NAMESPACE")

START_FROM = 1    # ← set to (last completed batch + 1) to resume

# ── Client ────────────────────────────────────────────────────────────────────
def get_client() -> VectorLakeClient:
    cfg = Config(
        api_key=API_KEY,
        host=HOST,
        vector_lake_path=VECTOR_LAKE_PATH,
    )
    return VectorLakeClient(cfg)

client = get_client()

# ── Print helper ──────────────────────────────────────────────────────────────
def _print(label: str, result) -> None:
    print(f"\n{'─'*70}")
    print(f"  {label}")
    print(f"{'─'*70}")
    if isinstance(result, list) and result and isinstance(result[0], dict):
        keys = list(result[0].keys())
        col  = max(len(k) for k in keys) + 2
        print("  " + "".join(k.ljust(col) for k in keys))
        print("  " + "─" * (col * len(keys)))
        for row in result:
            print("  " + "".join(str(row.get(k, "")).ljust(col) for k in keys))
    else:
        print(json.dumps(result, indent=2, default=str))

# ─────────────────────────────────────────────────────────────────────────────
#  UPLOAD / REFRESH / SYNC
# ─────────────────────────────────────────────────────────────────────────────

def run_health():
    _print("HEALTH CHECK", client.health_check(USER_ID, NAMESPACE))

def run_add_direct():
    """Direct mode: add brand-new files by passing content as strings."""
    result = client.add_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        files_name=["test1.txt", "test2.txt"],
        files_data=["hello world", "this is test doc 2"],
    )
    _print("ADD DOCUMENTS — direct", result)

def run_add_path():
    """
    Batch mode: add files from VECTOR_LAKE_PATH.
    Use only when ALL files are brand-new to the namespace.
    Use run_sync() for a mixed folder.
    """
    result = client.add_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM
    )
    _print(f"ADD DOCUMENTS — path (start={START_FROM})", result)

def run_refresh_direct():
    """Direct mode: refresh files that already exist in the namespace."""
    result = client.refresh_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        files_name=["test1.txt"],
        files_data=["UPDATED CONTENT FOR TEST1"],
    )
    _print("REFRESH DOCUMENTS — direct", result)

def run_refresh_path():
    """
    Batch mode: refresh files from VECTOR_LAKE_PATH.
    Use only when ALL files already exist in the namespace.
    Use run_sync() for a mixed folder.
    """
    result = client.refresh_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM,
    )
    _print(f"REFRESH DOCUMENTS — path (start={START_FROM})", result)

def run_sync():
    """
    *** RECOMMENDED FOR MOST USE CASES ***

    Classifies every file in VECTOR_LAKE_PATH and routes it automatically:
      No chunk on disk      → add_docs     (new file)
      Chunk exists, same    → skipped      (unchanged)
      Chunk exists, differs → refresh_docs (updated file)
    """
    result = client.sync_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM,
    )
    _print("SYNC DOCUMENTS", result)

def run_sync_dry_run():
    """Preview sync classification without touching the server."""
    result = client.sync_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        dry_run=True,
    )
    _print("SYNC DRY RUN — classification plan", result)

# ─────────────────────────────────────────────────────────────────────────────
#  QUERY
# ─────────────────────────────────────────────────────────────────────────────

def run_match_static(query: str):
    """Semantic search across ALL docs (search_type='flat')."""
    _print("MATCHING DOCS — flat (all docs)", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="static", hybrid_filter=True, search_type="flat",
        top_docs=5, threshold=0.1, with_data=True,
    ))

def run_match_filtered(query: str):
    """Semantic search ONLY over hybrid-filter-passing docs (search_type='flat_filter')."""
    _print("MATCHING DOCS — flat_filter (filtered docs only)", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="static", hybrid_filter=True, search_type="flat_filter",
        top_docs=5, threshold=0.1, with_data=True,
    ))

def run_match_dynamic(query: str):
    _print("MATCHING DOCS — dynamic / flat", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="dynamic", hybrid_filter=True, search_type="flat",
        files_name=["temp.txt"], files_data=["Sample dynamic content"],
    ))

def run_match_dynamic_filtered(query: str):
    _print("MATCHING DOCS — dynamic / flat_filter", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="dynamic", hybrid_filter=True, search_type="flat_filter",
        files_name=["temp.txt"], files_data=["Sample dynamic content"],
    ))

# ─────────────────────────────────────────────────────────────────────────────
#  INFO
# ─────────────────────────────────────────────────────────────────────────────

def run_namespace_details():
    _print("NAMESPACE DETAILS", client.get_namespace_details(USER_ID, NAMESPACE))

def run_docs_info():
    _print("DOCS INFORMATION", client.get_docs_information(USER_ID, NAMESPACE))

# ─────────────────────────────────────────────────────────────────────────────
#  MAIN — uncomment exactly ONE line
# ─────────────────────────────────────────────────────────────────────────────
if __name__ == "__main__":

    QUERY = """select top 2 where query is "what is cbc absolut count of yash patel"
     contains {yash patel} """

    # ── UPLOAD (choose based on server state) ────────────────────────────────
    # run_add_path()
    # run_add_direct()
    # run_refresh_path()
    # run_refresh_direct()
    # run_sync()           # ← recommended when folder has mix of new + existing files
    # run_sync_dry_run()   # ← preview sync plan without uploading

    # ── QUERY ────────────────────────────────────────────────────────────────
    # run_health()
    run_match_static(QUERY)           # flat        — all docs
    # run_match_filtered(QUERY)         # flat_filter — filtered docs only
    # run_match_dynamic(QUERY)
    # run_match_dynamic_filtered(QUERY)

    # ── INFO ─────────────────────────────────────────────────────────────────
    # run_namespace_details()
    # run_docs_info()
```

---

### Sample Outputs

#### `run_health()`

Pings the backend to confirm connectivity and that your namespace is reachable.

```
──────────────────────────────────────────────────────────────────────
  HEALTH CHECK
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "Processing time at server 0.00 ms",
    "docs": []
  },
  "message": "Health check successful",
  "status_code": 200
}
```

---

#### `run_add_direct()`

Uploads brand-new documents by passing file names and content directly as strings — no files on disk needed.

```
────────────────────────────────────────────────────────────
  ▶  ADD_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
──────────────────────────────────────────────────────────────────────
  ADD DOCUMENTS — direct
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "2 file(s) uploaded successfully",
    "docs": {
      "new_files": ["test2", "test1"],
      "updated_files": [],
      "failed_files": []
    }
  },
  "message": "2 file(s) uploaded successfully",
  "status_code": 200
}
```

---

#### `run_add_path()`

Reads all supported files from `VECTOR_LAKE_PATH`, chunks them into `{stem}_part{num}.txt` files, and uploads in batches with a live progress bar. Use only when **all files are brand-new** to the namespace.

```
────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS
────────────────────────────────────────────────────────────────
   Source files  : 152
   Chunks        : 164
   This run      : 39 batches
   Overall total : 39 batches
   Batch range   : #1 → #39
────────────────────────────────────────────────────────────────
add_docs:   3%|████                    | 1/39 [00:22<14:00, 22.1s/batch]
...
────────────────────────────────────────────────────────────────
  ■  ADD_DOCS COMPLETE
────────────────────────────────────────────────────────────────
   ✓  All batches succeeded
   Succeeded : 39/39
   Failed    : 0
────────────────────────────────────────────────────────────────
```

**Resuming after a crash** — set `START_FROM` to last successful batch + 1:
```python
START_FROM = 15   # batches 1–14 already done
```

---

#### `run_refresh_direct()`

Updates files that **already exist** in the namespace, passing updated content directly as strings.

```
────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "1 file(s) updated successfully",
    "docs": {
      "new_files": [],
      "updated_files": ["test1"],
      "failed_files": []
    }
  },
  "message": "1 file(s) updated successfully",
  "status_code": 200
}
```

---

#### `run_sync()` — Recommended

Classifies every file in the upload folder and automatically routes it.

```
────────────────────────────────────────────────────────────────
  🔍  CLASSIFYING FILES
────────────────────────────────────────────────────────────────
classify: 100%|████████████████████| 152/152 [00:02<00:00, file/s]

   NEW=12  CHANGED=5  UNCHANGED=135

────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS          (12 new files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCS      (5 changed files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ■  SYNC COMPLETE
────────────────────────────────────────────────────────────────
   Added     : 12
   Refreshed : 5
   Skipped   : 135
   Failed    : 0
────────────────────────────────────────────────────────────────
```

---

#### `run_sync_dry_run()`

Previews the sync plan without touching the server.

```
────────────────────────────────────────────────────────────────
  ■  DRY RUN COMPLETE — no files uploaded
────────────────────────────────────────────────────────────────
   NEW=12  REFRESH=5  SKIP=135
────────────────────────────────────────────────────────────────
{
  "mode": "dry_run",
  "plan": { "to_add": 12, "to_refresh": 5, "to_skip": 135, "total": 152 },
  "classifications": [
    { "filename": "report.pdf",  "status": "new",       "endpoint": "add_docs",     "reason": "No chunks on disk — first upload" },
    { "filename": "notes.txt",   "status": "changed",   "endpoint": "refresh_docs", "reason": "Content differs from existing chunks" },
    { "filename": "data.csv",    "status": "unchanged", "endpoint": "skip",         "reason": "Content matches existing chunks — skipping" }
  ]
}
```

---

#### `run_match_static(query)` — `search_type="flat"`

Full-corpus fusion search across ALL documents.

```
──────────────────────────────────────────────────────────────────────
  MATCHING DOCS — flat (all docs)
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "file_name": "electrolytes-test-report-f_part1",
        "doc_score": 199.95,
        "category": "Very Good Match",
        "chunks": {
          "chunk": ["Yash M. Patel Age: 21 Years ... Sodium 110.00 Low ..."],
          "similarities": [2.0]
        }
      }
    ]
  },
  "message": "Data fetched successfully",
  "status_code": 200
}
```

---

#### `run_match_filtered(query)` — `search_type="flat_filter"`

Precision search — semantic ranking only runs over filter-passing documents.

**Sample output:** Identical structure to `flat` above. The difference is in the candidate pool — results are narrower and more precise.

---

#### `run_match_dynamic(query)` / `run_match_dynamic_filtered(query)`

Same as static search but over **temporary files provided at query time** — nothing needs to be pre-indexed.

---

#### `run_namespace_details()`

```
──────────────────────────────────────────────────────────────────────
  NAMESPACE DETAILS
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "vector_lake_description": "customdataset_",
        "faiss_disk_size_mb": "14.43 mb",
        "source_store_files": 166,
        "source_store_used_mb": "30.9062 mb",
        "source_store_quota_remaining_mb": "12257.09 mb"
      }
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}
```

---

#### `run_docs_info()`

```
──────────────────────────────────────────────────────────────────────
  DOCS INFORMATION
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "docs": [
      "001-hide-and-seek-free-chi",
      "002-ginger-the-giraffe-fre",
      "2._a_tale_of_two_cities_au",
      ...
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}
```

---

## 📚 VectorLake Client API Reference

All public methods return a plain `dict` and **never raise** — errors are surfaced as `{"success": False, "error": "...", "message": "..."}`.

### 1. `add_documents`

Uploads **new** documents to the index. The server rejects files already present in the namespace — use `sync_documents()` for mixed folders.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier (registration email). |
| `vector_lake_description` | str | required | Target namespace. |
| `files_name` + `files_data` | List[str] | — | **Direct mode**: names and string contents. Both required together, equal length. |
| `files` | List[str] | `None` | **Batch mode**: filenames to read from `vector_lake_path`. Defaults to all supported files. |
| `start_from_batch` | int | `1` | Resume point for batch mode. |
| `end_batch` | int | `None` | Upper batch bound (inclusive). `None` = process all. |
| `intelligent_segmentation` | bool | `True` | Enables server-side segmentation before embedding. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 2. `refresh_documents`

Updates **existing** documents in the index. Same parameters and modes as `add_documents`. The server rejects files that do not already exist. Use `sync_documents()` for mixed folders.

---

### 3. `sync_documents`

Intelligently syncs a folder by routing each file to the correct endpoint based on MD5 comparison against existing chunks on disk.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Target namespace. |
| `files` | List[str] | `None` | Files to sync. Defaults to all supported files in `vector_lake_path`. |
| `dry_run` | bool | `False` | If `True`, returns classification plan without uploading. |
| `start_from_batch` | int | `1` | Resume point. |
| `end_batch` | int | `None` | Upper batch bound. |
| `intelligent_segmentation` | bool | `True` | Server-side segmentation. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 4. `get_matching_docs`

Retrieves top-matching document chunks using semantic search with optional hybrid filtering.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | str | required | Natural language or WaveQL search query. |
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Target namespace. |
| `pattern` | str | `"static"` | `"static"` for indexed docs; `"dynamic"` for temporary files. |
| `hybrid_filter` | bool | `False` | Enables keyword hybrid filter. **Required when `search_type` is set.** |
| `search_type` | str | `None` | `"flat"` — full corpus fusion. `"flat_filter"` — semantic search over filter-passing docs only. `None` — server default. |
| `top_docs` | int | `10` | Maximum chunks to return. |
| `threshold` | float | `0.2` | Similarity score cutoff. |
| `with_data` | bool | `False` | If `True`, includes raw chunk text in response. |
| `files_name` + `files_data` | List[str] | `None` | Dynamic mode: temporary files to search over. |
| `session_id` | str | `None` | Optional session identifier. |

**Validation rules:**
- `search_type` must be `"flat"` or `"flat_filter"`; anything else returns `InvalidSearchTypeError`.
- `hybrid_filter=True` is required whenever `search_type` is specified.

---

### 5. `health_check`

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to check. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 6. `get_namespace_details`

Returns storage and quota metadata for one or all namespaces belonging to the user.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | `None` | Scopes response to that namespace if supplied. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 7. `get_docs_information`

Returns document-level metadata within a namespace, optionally filtered by keyword.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to query. |
| `keyword` | str | `None` | Optional filter keyword. |
| `threshold` | int | `70` | Keyword-match threshold. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 8. `full_corpus_search`

Full-text keyword search across all documents in a namespace.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to search. |
| `keyword` | str | required | Search term. |
| `top_docs` | int | `10` | Maximum results to return. |
| `session_id` | str | `None` | Optional session identifier. |

---

##  Error Handling

All exceptions inherit from `VectorLakeError` and expose `.to_response()`. Since all public methods catch and convert exceptions internally, you only ever need to check the `"error"` key in the returned dict.

| Exception | When raised |
|-----------|------------|
| `ConfigError` | API key missing or config invalid at init time. |
| `ValidationError` | Mismatched `files_name`/`files_data` lengths or other pre-flight failures. |
| `InvalidSearchTypeError` | `search_type` is not `"flat"` or `"flat_filter"`. |
| `UnsupportedFileTypeError` | File extension not in `allowed_extensions`. |
| `FileProcessingError` | File I/O, encoding, or parsing failure. |
| `APIError` | HTTP 4xx/5xx from server. Carries `.status_code` and `.response_text`. |
| `ThrottleError` | HTTP 429 rate limit. Carries `.retry_after`. |

---

##  Structured Logging

The `Logger` class writes three CSV files to `log_dir` (default: `logs/`):

| File | Contents |
|------|----------|
| `performance.csv` | Per-request latency, payload/response KB, HTTP status |
| `api_errors.csv` | Operation, batch number, error message |
| `skipped_files.csv` | Filename and reason when a file is skipped |

---

##  Retry and Backoff

```
  Request attempt 1
      │
      ├── HTTP 429     → wait Retry-After header (or 2^attempt s) → retry
      ├── Timeout      → wait 2^attempt s → retry
      ├── ConnError    → wait 2^attempt s → retry
      ├── HTTP 4xx/5xx → return error dict immediately (no retry)
      └── Success      → return response dict
```

Max retries: 2 (configurable via `VECTOR_LAKE_MAX_RETRIES`).

---

##  Using WaveQL (VQL) Queries

WaveQL enables natural language filtering using brace-based logical groups.

### Syntax at a Glance

```
  ┌──────────────────────────────────────────────────────────────────┐
  │  {(clinical trials) or (observational studies)} {diabetes}       │
  │   └──────────────── group 1 ─────────────────┘  └── group 2 ──┘ │
  │                                                                  │
  │  Groups combine with implicit AND:                               │
  │  → (clinical trials OR observational studies) AND diabetes       │
  └──────────────────────────────────────────────────────────────────┘

  Within a group:
  ┌──────────────────────────────────────────────────────────┐
  │  {A and B}       → both A and B must match               │
  │  {A or B}        → either A or B                         │
  │  {(A B) or C}    → phrase "A B", or term C               │
  │  {A B}           → implicit AND: A and B                 │
  └──────────────────────────────────────────────────────────┘

  ✅ Correct                        ❌ Wrong
  ──────────────────────────────    ─────────────────────────────
  {(machine learning) or            {machine learning or
   (deep learning)}                  deep learning}
                                      → "machine" treated as
                                        a separate term

  {(product manager) or             {product manager or Delhi}
   (data scientist)}                  → ambiguous parse
```

### Three-Tier Result Ranking

```
  ┌─────────────────────────────────────────────────────────────┐
  │             RESULT TIERS (highest → lowest)                 │
  ├──────────┬──────────────────────────────────────────────────┤
  │  Tier 1  │  ✓ Filter match  +  ✓ Semantic match             │
  │          │  Highest confidence — structure & meaning align  │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 2  │  ✓ Filter match  +  ✗ Semantic match             │
  │          │  Structured match, lower semantic relevance      │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 3  │  ✗ Filter match  +  ✓ Semantic match             │
  │          │  Meaning-based fallback, no filter alignment     │
  └──────────┴──────────────────────────────────────────────────┘
```

### Query Examples by Domain

**Healthcare:**
```
Select Top 10 
Where QUERY IS "Need detail about disease state progression of patient id 555"
Contains {diabetes} {(clinical trial)} {PID 555}
```

**Recruitment:**
```
Select Top 30 
Where QUERY IS "Need list of resources who have good experience in python and machine learning"
Contains {Python} {(machine learning)} {Delhi}
```

### Filter Design Best Practices

**DO:** Use 1–2 keywords per brace · Wrap multi-word phrases in parentheses with operators · Keep filters domain-consistent · Trust semantic fallback for edge cases

**DON'T:** Use 5+ word phrases · Mix unrelated domains (`{resume} {clinical trials}`) · Forget parentheses around multi-word phrases with OR/AND · Over-specify filters

---

## 🎯 No Schema Required

```
  Traditional approach                  WaveflowDB approach
  ────────────────────                  ───────────────────
  1. Define JSON schema           vs.   1. Upload raw files
  2. Extract & map every field             (PDF, txt, docx, csv…)
  3. Maintain schema consistency
  4. Update schema for new fields   →   2. Query immediately
  5. Re-index on schema change             with WaveQL
```

| Feature | Traditional | WaveflowDB |
|---------|-------------|------------|
| Data Ingestion | Extract, map, validate | Direct upload |
| Schema Definition | Required upfront | Not required |
| Query Capability | Exact field matching | Semantic + logical filtering |
| New Document Types | Requires schema update | Works immediately |
| Maintenance | High | Low |

---

## 📧 Support

For API or platform support, visit: **https://db.agentanalytics.ai**

---

## 📄 License

Copyright DIBR tech private ltd.
