Metadata-Version: 2.4
Name: tdfs4ds
Version: 0.3.1.166
Summary: A python package to simplify the usage of feature store using Teradata Vantage ...
Author: Denis Molin
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: teradataml>=17.20
Requires-Dist: pandas
Requires-Dist: numpy
Requires-Dist: plotly
Requires-Dist: tqdm
Requires-Dist: networkx
Requires-Dist: sqlparse
Requires-Dist: langchain_openai
Requires-Dist: langchain_aws
Requires-Dist: langchain_core
Requires-Dist: langchain_chroma
Requires-Dist: langchain_teradata
Requires-Dist: langchain_mistralai
Requires-Dist: langgraph
Requires-Dist: chromadb
Requires-Dist: pydantic
Requires-Dist: gradio
Requires-Dist: nbformat>=4.2.0
Requires-Dist: langchain_mcp_adapters
Requires-Dist: pyvis
Requires-Dist: teradatagenai
Requires-Dist: pysqlite3-binary; platform_system == "Linux"
Provides-Extra: serve
Requires-Dist: fastapi; extra == "serve"
Requires-Dist: uvicorn[standard]; extra == "serve"
Requires-Dist: langserve[server]; extra == "serve"
Requires-Dist: sse-starlette; extra == "serve"
Requires-Dist: httpx[socks]; extra == "serve"
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

![tdfs4ds logo](https://github.com/denismolin/tdfs4ds/raw/main/tdfs4ds_logo.png)

# tdfs4ds — A Feature Store Library for Data Scientists working with ClearScape Analytics

`tdfs4ds` (Teradata Feature Store for Data Scientists) is a Python package for managing temporal feature stores in Teradata Vantage databases. It provides easy-to-use functions for creating, registering, storing, and retrieving features — with full time-travel support, lineage tracking, and process operationalization.

## Installation

```bash
pip install tdfs4ds
```

## Quick Start

Import `tdfs4ds` **after** establishing a teradataml connection so the package can auto-detect your default database:

```python
import teradataml as tdml
tdml.create_context(host=..., username=..., password=...)

import tdfs4ds
# tdfs4ds.SCHEMA is auto-set from the teradataml context;
# override if needed: tdfs4ds.SCHEMA = 'my_database'

# Data domain management — use the dedicated functions:
tdfs4ds.create_data_domain('MY_PROJECT')   # create and activate a new domain
# or
tdfs4ds.select_data_domain('MY_PROJECT')   # activate an existing domain
# or
tdfs4ds.get_data_domains()                 # list all available domains (* marks the active one)
```

## Core API

| Function | Description |
|----------|-------------|
| `tdfs4ds.setup(database)` | Create feature catalog, process catalog, and follow-up tables in `database` |
| `tdfs4ds.upload_features(df, entity_id, feature_names, metadata={})` | Ingest features from a teradataml DataFrame into the feature store |
| `tdfs4ds.build_dataset(entity_id, selected_features, view_name, comment='dataset', grouped=False)` | Assemble a dataset view from registered features |
| `tdfs4ds.run(process_id)` | Re-execute a registered feature engineering process |
| `tdfs4ds.roll_out(...)` | Operationalize processes at scale |
| `tdfs4ds.connect(database, databases_readonly=[...])` | Connect to a feature store; optionally extend reads to other stores (read-only) |
| `tdfs4ds.extend_read_stores([...])` / `detach_read_stores()` / `list_read_stores()` | Attach / detach / list read-only feature stores (cross-store read federation) |

### `entity_id` must specify SQL data types (dict, not list)

```python
entity_id = {'CUSTOMER_ID': 'BIGINT', 'EVENT_DATE': 'DATE'}   # correct
entity_id = ['CUSTOMER_ID', 'EVENT_DATE']                      # wrong
```

## Walkthrough Example

### Step 1 — Set up a feature store

```python
import teradataml as tdml
tdml.create_context(host=..., username=..., password=...)

import tdfs4ds
tdfs4ds.setup(database='my_database')
```

### Step 2 — Configure the active context

```python
tdfs4ds.SCHEMA = 'my_database'   # override if not auto-detected

# Use dedicated functions to manage the data domain:
tdfs4ds.create_data_domain('DATA_QUALITY')   # create and activate (first time)
# tdfs4ds.select_data_domain('DATA_QUALITY') # activate an existing domain
# tdfs4ds.get_data_domains()                 # list all domains
```

### Step 3 — Define your feature engineering view

```python
df = tdml.DataFrame(tdml.in_schema('my_database', 'my_feature_view'))
# If teradataml created intermediate views, make them permanent first:
# tdfs4ds.crystallize_view(df)
```

### Step 4 — Upload and operationalize

```python
entity_id     = {'EVENT_DT': 'DATE', 'ID': 'BIGINT'}
feature_names = ['KPI1', 'KPI2']

tdfs4ds.upload_features(
    df=df,
    entity_id=entity_id,
    feature_names=feature_names,
    metadata={'project': 'data quality'}
)
```

This registers entities and features (if not already present), registers a feature engineering process in the process catalog, and writes the feature values into the feature store.

> **UAF features (secondary ART layers).** For a feature whose value lives in a
> secondary Teradata UAF ART layer — ARIMA fit-metrics (`R_SQUARE`/`MAPE`/`AIC` in
> `ARTFITMETADATA`), residual tests over `ARTFITRESIDUALS`, `TD_PACF` — pass
> `art_sql=[...]` (the `INTO VOLATILE ART` statements) and `art_layers=[...]` to
> `upload_features`. The process is registered as `PROCESS_TYPE='uaf art view'` and the
> ART is materialised per partition, in-session, before each ingestion. See the
> [UAF ART-layer features guide](https://denismolin.github.io/tdfs4ds/user-guide/uaf-art-view/).

### Step 5 — Re-run a process

```python
# List all registered processes to find the process ID
tdfs4ds.process_catalog()

# Re-execute by process ID
tdfs4ds.run(process_id)
```

### Step 6 — Build a dataset

```python
selected_features = {
    'KPI1': '<process_uuid>',
    'KPI2': '<process_uuid>',
}

dataset = tdfs4ds.build_dataset(
    entity_id={'ID': 'BIGINT'},
    selected_features=selected_features,
    view_name='my_dataset',
    comment='Dataset for churn model'
)
```

`selected_features` maps each feature name to the UUID of the process that computed it.

Pass `grouped=True` to activate the grouped pivot strategy — when many features share the same source table and process, they are collapsed into a single `MAX(CASE WHEN FEATURE_ID=…) … GROUP BY` sub-query instead of one sub-query per feature. This reduces JOIN fan-out for wide feature sets:

```python
dataset = tdfs4ds.build_dataset(
    entity_id={'ID': 'BIGINT'},
    selected_features=selected_features,
    view_name='my_dataset',
    comment='Dataset for churn model',
    grouped=True,   # MAX(CASE WHEN) + GROUP BY pivot strategy
)
```

Use `return_query=True` to inspect the generated DDL without executing it:

```python
sql = tdfs4ds.build_dataset(
    entity_id={'ID': 'BIGINT'},
    selected_features=selected_features,
    view_name='my_dataset',
    return_query=True,
)
print(sql)
```

## Configuration

### Programmatic (in-session)

```python
tdfs4ds.SCHEMA                = 'my_database'        # target database (auto-set from context)
# Data domain: use tdfs4ds.create_data_domain() / select_data_domain() / get_data_domains()
tdfs4ds.FEATURE_STORE_TIME    = None                 # None = current; '2024-01-01 00:00:00' = time travel
tdfs4ds.DISPLAY_LOGS          = True                 # verbose logging
tdfs4ds.DEBUG_MODE            = False
tdfs4ds.STORE_FEATURE         = 'MERGE'              # 'MERGE' or 'UPDATE_INSERT'

# GenAI documentation (instruct models)
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'openai'           # or 'bedrock', 'vllm', 'openai-compatible', 'azure', 'mistralai', 'gcp'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'gpt-4o'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = 'sk-...'           # prefer env var instead (see below)
tdfs4ds.INSTRUCT_MODEL_REGION   = None               # AWS region for Bedrock (e.g., 'us-east-1')
tdfs4ds.INSTRUCT_MODEL_AZURE_DEPLOYMENT = None       # Azure deployment name (optional)
tdfs4ds.INSTRUCT_MODEL_AZURE_API_VERSION = '2024-02-01'
tdfs4ds.INSTRUCT_MODEL_GCP_PROJECT = None            # GCP project ID for VertexAI
tdfs4ds.INSTRUCT_MODEL_GCP_LOCATION = 'us-central1'  # GCP region
# TLS for instruct calls only (embeddings are never affected):
tdfs4ds.INSTRUCT_MODEL_CA_BUNDLE  = None             # internal CA PEM; verification stays ON
tdfs4ds.INSTRUCT_MODEL_VERIFY_SSL = True             # False disables verification (last resort)

# Embedding model (consumer agent vector index — falls back to INSTRUCT_MODEL_* if unset)
tdfs4ds.EMBEDDING_MODEL_PROVIDER = 'vllm'
tdfs4ds.EMBEDDING_MODEL_URL      = 'https://api.example.com/v1/e5'
tdfs4ds.EMBEDDING_MODEL_MODEL    = 'text-embedding-3-small'
tdfs4ds.EMBEDDING_MODEL_DIM      = 1536
tdfs4ds.EMBEDDING_MODEL_REGION   = None               # Falls back to INSTRUCT_MODEL_REGION
tdfs4ds.EMBEDDING_MODEL_AZURE_DEPLOYMENT = None
tdfs4ds.EMBEDDING_MODEL_GCP_PROJECT = None

# Chroma vector store
tdfs4ds.CHROMA_MODE = 'local'                        # 'local' or 'server'
tdfs4ds.CHROMA_PATH = './tdfs4ds_chroma'             # persist directory (local mode)

# MCP server (optional — consumer agent external tools)
tdfs4ds.MCP_SERVER_URL = 'http://localhost:8000/sse' # SSE endpoint; None = disabled
```

### Config file (persistent per-project or per-user)

Create a `tdfs4ds.json` file in your project directory (or `~/.tdfs4ds/config.json` for user-wide defaults) to avoid repeating the setup cell in every notebook:

```json
{
    "schema": "MY_DATABASE",
    "data_domain": "MY_PROJECT",
    "display_logs": true,
    "store_feature": "MERGE",
    "varchar_size": 1024,
    "instruct_model_provider": "openai",
    "instruct_model_model": "gpt-4o",
    "instruct_model_url": null,
    "embedding_model_provider": "vllm",
    "embedding_model_model": "text-embedding-3-small",
    "embedding_model_dim": 1536,
    "chroma_mode": "local",
    "chroma_path": "./tdfs4ds_chroma",
    "mcp_server_url": null
}
```

Keys are case-insensitive. `instruct_model_api_key` is rejected from JSON config to prevent accidental commits — use a `.env` file or OS env var for credentials.

### `.env` file (local secrets and overrides)

Place a `.env` file in your project directory (or `~/.tdfs4ds/.env` for user-wide defaults). Only `TDFS4DS_*` variables are read — the file is parsed without touching `os.environ`:

```dotenv
TDFS4DS_SCHEMA=MY_DATABASE
TDFS4DS_DATA_DOMAIN=MY_PROJECT
TDFS4DS_INSTRUCT_MODEL_API_KEY=sk-...
TDFS4DS_INSTRUCT_MODEL_PROVIDER=openai
TDFS4DS_INSTRUCT_MODEL_MODEL=gpt-4o
TDFS4DS_INSTRUCT_MODEL_REGION=us-east-1
TDFS4DS_INSTRUCT_MODEL_AZURE_DEPLOYMENT=my-deployment
TDFS4DS_INSTRUCT_MODEL_GCP_PROJECT=my-gcp-project
TDFS4DS_EMBEDDING_MODEL_PROVIDER=vllm
TDFS4DS_EMBEDDING_MODEL_URL=https://api.example.com/v1/e5
TDFS4DS_EMBEDDING_MODEL_MODEL=text-embedding-3-small
TDFS4DS_EMBEDDING_MODEL_DIM=1536
TDFS4DS_CHROMA_MODE=local
TDFS4DS_CHROMA_PATH=./tdfs4ds_chroma
TDFS4DS_MCP_SERVER_URL=https://your-mcp-server/endpoint/
TDFS4DS_MCP_SERVER_TRANSPORT=streamable_http
# TDFS4DS_MCP_SERVER_EDITION=enterprise   # 'community' (default) sends no credentials

# Serve (HTTP server / Docker — tdfs4ds[serve] only)
TDFS4DS_TD_HOST=your-vantage-host
TDFS4DS_TD_USERNAME=your_user
TDFS4DS_TD_PASSWORD=your_password
# TDFS4DS_TD_LOGMECH=LDAP
# TDFS4DS_TD_DATABASE=your_db
# TDFS4DS_TD_ENCRYPT=true

# Vector Store authentication (only needed for VECTOR_STORE_BACKEND='teradata')
# TDFS4DS_VECTOR_STORE_AUTH_BASE_URL=https://your-vantage-vs-endpoint
# TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN=...      # JWT — a token you already hold (tried first);
#                                               # in-cluster, map the platform's injected token
#                                               # onto it, e.g. =$OAUTH_ACCESS_TOKEN
# TDFS4DS_VECTOR_STORE_AUTH_PAT_TOKEN=...       # PAT
# TDFS4DS_VECTOR_STORE_AUTH_PEM_FILE=/path/to/key.pem
# TDFS4DS_VECTOR_STORE_AUTH_USERNAME=your_user  # optional for PAT, required for Basic
# TDFS4DS_VECTOR_STORE_AUTH_PASSWORD=...        # Basic
# TDFS4DS_VECTOR_STORE_AUTH_CLIENT_ID=...       # OAuth

TDFS4DS_SERVE_PORT=8000
TDFS4DS_SERVE_BUILD_INDEX=true
TDFS4DS_SERVE_CHATBOT=true
TDFS4DS_SERVE_CHATBOT_PORT=7860
```

Add `.env` to `.gitignore` to keep secrets out of source control. Quoted values and `export KEY=VALUE` syntax are supported.

### Environment variables

All settings can also be set via `TDFS4DS_<VAR_NAME>` OS environment variables (useful in CI/CD):

| Variable | Corresponding setting |
|---|---|
| `TDFS4DS_SCHEMA` | `tdfs4ds.SCHEMA` |
| `TDFS4DS_SCHEMA_READONLY` | `tdfs4ds.SCHEMA_READONLY` (comma-separated list of read-only feature stores) |
| `TDFS4DS_DATA_DOMAIN` | `tdfs4ds.DATA_DOMAIN` |
| `TDFS4DS_DISPLAY_LOGS` | `tdfs4ds.DISPLAY_LOGS` |
| `TDFS4DS_DEBUG_MODE` | `tdfs4ds.DEBUG_MODE` |
| `TDFS4DS_STORE_FEATURE` | `tdfs4ds.STORE_FEATURE` |
| `TDFS4DS_VARCHAR_SIZE` | `tdfs4ds.VARCHAR_SIZE` |
| `TDFS4DS_INSTRUCT_MODEL_PROVIDER` | `tdfs4ds.INSTRUCT_MODEL_PROVIDER` |
| `TDFS4DS_INSTRUCT_MODEL_MODEL` | `tdfs4ds.INSTRUCT_MODEL_MODEL` |
| `TDFS4DS_INSTRUCT_MODEL_URL` | `tdfs4ds.INSTRUCT_MODEL_URL` |
| `TDFS4DS_INSTRUCT_MODEL_API_KEY` | `tdfs4ds.INSTRUCT_MODEL_API_KEY` |
| `TDFS4DS_INSTRUCT_MODEL_REGION` | `tdfs4ds.INSTRUCT_MODEL_REGION` |
| `TDFS4DS_INSTRUCT_MODEL_VERIFY_SSL` | `'false'` disables TLS verification for instruct calls — last resort, prefer the CA bundle |
| `TDFS4DS_INSTRUCT_MODEL_CA_BUNDLE` | Path to an internal CA PEM for instruct calls; verification stays enabled |
| `TDFS4DS_INSTRUCT_MODEL_AZURE_DEPLOYMENT` | `tdfs4ds.INSTRUCT_MODEL_AZURE_DEPLOYMENT` |
| `TDFS4DS_INSTRUCT_MODEL_AZURE_API_VERSION` | `tdfs4ds.INSTRUCT_MODEL_AZURE_API_VERSION` |
| `TDFS4DS_INSTRUCT_MODEL_GCP_PROJECT` | `tdfs4ds.INSTRUCT_MODEL_GCP_PROJECT` |
| `TDFS4DS_INSTRUCT_MODEL_GCP_LOCATION` | `tdfs4ds.INSTRUCT_MODEL_GCP_LOCATION` |
| `TDFS4DS_EMBEDDING_MODEL_PROVIDER` | `tdfs4ds.EMBEDDING_MODEL_PROVIDER` |
| `TDFS4DS_EMBEDDING_MODEL_MODEL` | `tdfs4ds.EMBEDDING_MODEL_MODEL` |
| `TDFS4DS_EMBEDDING_MODEL_URL` | `tdfs4ds.EMBEDDING_MODEL_URL` |
| `TDFS4DS_EMBEDDING_MODEL_API_KEY` | `tdfs4ds.EMBEDDING_MODEL_API_KEY` |
| `TDFS4DS_EMBEDDING_MODEL_DIM` | `tdfs4ds.EMBEDDING_MODEL_DIM` |
| `TDFS4DS_EMBEDDING_MODEL_REGION` | `tdfs4ds.EMBEDDING_MODEL_REGION` |
| `TDFS4DS_EMBEDDING_MODEL_AZURE_DEPLOYMENT` | `tdfs4ds.EMBEDDING_MODEL_AZURE_DEPLOYMENT` |
| `TDFS4DS_EMBEDDING_MODEL_GCP_PROJECT` | `tdfs4ds.EMBEDDING_MODEL_GCP_PROJECT` |
| `TDFS4DS_CHROMA_MODE` | `tdfs4ds.CHROMA_MODE` |
| `TDFS4DS_CHROMA_PATH` | `tdfs4ds.CHROMA_PATH` |
| `TDFS4DS_CHROMA_HOST` | `tdfs4ds.CHROMA_HOST` |
| `TDFS4DS_CHROMA_PORT` | `tdfs4ds.CHROMA_PORT` |
| `TDFS4DS_MCP_SERVER_URL` | `tdfs4ds.MCP_SERVER_URL` |
| `TDFS4DS_MCP_SERVER_TRANSPORT` | `tdfs4ds.MCP_SERVER_TRANSPORT` (`'streamable_http'` or `'sse'`) |
| `TDFS4DS_MCP_SERVER_EDITION` | `'community'` (default, sends no credentials) or `'enterprise'` (HTTP Basic) |
| `TDFS4DS_MCP_AUTH` | Override the mechanism the edition implies — `'none'` or `'basic'` |
| `TDFS4DS_MCP_USERNAME` | MCP user — falls back to `TDFS4DS_TD_USERNAME` |
| `TDFS4DS_MCP_PASSWORD` | MCP password — falls back to `TDFS4DS_TD_PASSWORD` |
| `TDFS4DS_MCP_SERVER_HEADERS` | JSON object of verbatim headers (Bearer, API key…); wins over the above |
| `TDFS4DS_MCP_SERVER_CA_BUNDLE` | Path to an internal CA PEM; TLS verification stays enabled |
| `TDFS4DS_SKILLS_FOLDER` | Path to a folder containing plugin consumer-agent skill subdirectories |
| **Teradata connection (`tdfs4ds.serve` only)** | |
| `TDFS4DS_TD_HOST` | Teradata host — required by the HTTP server bootstrap |
| `TDFS4DS_TD_USERNAME` | Teradata user — required by the HTTP server bootstrap |
| `TDFS4DS_TD_PASSWORD` | Teradata password — required by the HTTP server bootstrap |
| `TDFS4DS_TD_LOGMECH` | Logon mechanism (e.g. `LDAP`, `TD2`) — optional |
| `TDFS4DS_TD_DATABASE` | Default database for the session — optional |
| `TDFS4DS_TD_ENCRYPT` | Wire encryption — `'true'` or `'false'` — optional |
| **Vector Store authentication (`VECTOR_STORE_BACKEND='teradata'` only)** — see [Vector Store authentication](#vector-store-authentication) | |
| `TDFS4DS_VECTOR_STORE_AUTH_BASE_URL` | Vector Store REST endpoint URL — required by all four mechanisms below |
| `TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN` | A token you already hold (JWT auth, tried first) |
| `TDFS4DS_VECTOR_STORE_AUTH_PAT_TOKEN` | PAT token — used with `PEM_FILE` (PAT auth) |
| `TDFS4DS_VECTOR_STORE_AUTH_PEM_FILE` | Path to the PAT private key file — used with `PAT_TOKEN` |
| `TDFS4DS_VECTOR_STORE_AUTH_USERNAME` | Username — optional for PAT, required for Basic auth |
| `TDFS4DS_VECTOR_STORE_AUTH_PASSWORD` | Password — Basic auth (fallback if PAT is not fully configured) |
| `TDFS4DS_VECTOR_STORE_AUTH_CLIENT_ID` | OAuth client id (fallback if no other group is fully configured) |
| **Server behaviour (`tdfs4ds.serve` only)** | |
| `TDFS4DS_SERVE_PORT` | API server port (default `8000`) |
| `TDFS4DS_SERVE_BUILD_INDEX` | `'true'` (default) builds vector index at startup; `'false'` for QO-only deployments |
| `TDFS4DS_SERVE_CHATBOT` | `'true'` (default) launches the Gradio admin chatbot in the same process |
| `TDFS4DS_SERVE_CHATBOT_PORT` | Admin chatbot port (default `7860`) |

### Vector Store authentication

Only relevant when you use `tdfs4ds.VECTOR_STORE_BACKEND = 'teradata'` (the
Teradata Enterprise Vector Store, instead of the default local Chroma index).

The Vector Store is reached over a **REST API**, which authenticates through
`teradataml.set_auth_token(...)` — a **different channel** from the
`tdml.create_context()` database session. `setup()`, `connect()` and the
`tdfs4ds.serve` server all configure it automatically and non-fatally from
environment variables: set `TDFS4DS_VECTOR_STORE_AUTH_BASE_URL` plus **one**
credential group. The first fully-configured group wins, in this order:

| Priority | Mechanism | Additional variables |
|---|---|---|
| 1 | **JWT** — a token you already hold | `TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN` |
| 2 | **PAT** | `..._PAT_TOKEN` + `..._PEM_FILE` (+ optional `..._USERNAME`) |
| 3 | **Basic** | `..._USERNAME` + `..._PASSWORD` |
| 4 | **OAuth** | `..._CLIENT_ID` |

PAT, Basic and OAuth make `tdfs4ds` *generate* a token. **JWT** uses one you
supply, which is why it is tried first: an explicitly provided token should
never be silently ignored in favour of a generated one.

**In-cluster (Kubernetes / VantageCloud Lake), the token is usually already in
the pod** — injected by the platform under its own name. Map it onto the JWT
variable:

```bash
TDFS4DS_VECTOR_STORE_AUTH_BASE_URL=http://vectorstore.vectorstore.svc.cluster.local:8000
TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN=$OAUTH_ACCESS_TOKEN
```

That is the equivalent of doing it by hand:

```python
tdml.set_auth_token(base_url='http://vectorstore.vectorstore.svc.cluster.local:8000',
                    auth_token=os.getenv('OAUTH_ACCESS_TOKEN'))
```

Two things worth knowing:

- **Only `TDFS4DS_`-prefixed variables are read.** `tdfs4ds` never picks up an
  unprefixed environment variable for authentication, however conventional the
  name — a generic name may be owned by another component in the same
  environment, and reading it would send that component's credential to the
  Vector Store endpoint with nothing in your configuration naming it. Hence the
  explicit mapping above.
- The mechanism used is logged at `connect()`/`setup()` time; **the token value
  never is** — only the `base_url` and the *name* of the variable it came from.

In that same in-cluster setup the database session itself is typically opened
with a JWT logon — `tdml.create_context(host=..., logmech='JWT', logdata=...)`
— which is a separate concern from the Vector Store REST auth above.

### Checking your configuration

`tdfs4ds.check_configuration()` validates the AI-facing configuration and, for
anything that does not pass, names the setting to change:

```python
import tdfs4ds
tdfs4ds.check_configuration()             # real handshake, one prompt, one embedding
tdfs4ds.check_configuration(live=False)   # resolve config + build clients, call nothing
```

```text
tdfs4ds configuration check — v0.3.1.154

✅ Vector Store auth       JWT (base_url=http://vectorstore…:8000 token from TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN)
✅ Vector Store service    healthy
✅ Instruct model          openai / gpt-4o — replied 'hello world' in 0.61s
❌ Embedding model         openai / text-embedding-3-small — embedded 'hello world' into 1024 dimensions
                           → EMBEDDING_MODEL_DIM is 1536 but the model returns 1024
                           fix: Set tdfs4ds.EMBEDDING_MODEL_DIM = 1024 (or
                                TDFS4DS_EMBEDDING_MODEL_DIM=1024). A mismatch here does not
                                raise — it silently builds a vector index that cannot be searched.
```

Six checks run, in dependency order:

| Check | Applies when | What it does |
|---|---|---|
| **Config load** | always | Compares the `TDFS4DS_*` environment against the module config — see below |
| **Vector Store auth** | backend is `'teradata'`, or any auth variable is set | Resolves and applies `set_auth_token`, reporting the mechanism |
| **Vector Store service** | backend is `'teradata'` | `CollectionManager().health()` |
| **Instruct model** | `INSTRUCT_MODEL_PROVIDER` + `_MODEL` set | Builds the client with `build_llm()`, then sends a hello-world prompt |
| **Embedding model** | always | Branches on the backend — see below |
| **MCP server** | `MCP_SERVER_URL` set | Lists the tools, then asks a model to call one — see below |

The **MCP** check is two stages, and which one fails is the diagnosis. Listing
tools exercises the URL, the credentials and TLS; the tool-calling probe
exercises the *model*, since an endpoint can chat perfectly and still not
forward tool schemas. A stage-1 failure is therefore never caused by a broken
instruct model — without one, stage 2 is **skipped**, not failed. Override the
probe question with `check_configuration(mcp_question='…')`; it defaults to
listing databases, which both editions can answer.

The embedding check differs per backend on purpose. With the **Teradata Vector
Store** and `TERADATA_AI_API_TYPE` set, it verifies only that `TeradataAI(...)`
**instantiates** — in-database embedding needs nothing further from the client.
With **ChromaDB** (or the Teradata backend without `TERADATA_AI_API_TYPE`, which
falls back to LangChain embeddings) it embeds a string and checks the returned
vector length against `EMBEDDING_MODEL_DIM` — a mismatch there never raises, it
just silently produces an index that cannot be searched.

Every check that fails — or is skipped for want of a setting — carries a `fix:`
hint. The function **never raises** and **never prints a secret** (only variable
*names*, URLs and model names), and returns a dict so it can gate a startup
script:

```python
report = tdfs4ds.check_configuration(display=False)
if not report["ok"]:
    failed = [c["label"] for c in report["checks"] if c["status"] == "failed"]
    raise SystemExit(f"tdfs4ds is not correctly configured: {', '.join(failed)}")
```

The same check is in the admin UI under **Parameters → Configuration check**.

The Teradata **database** connection is deliberately not checked — it is
external to the platform and opened by you.

#### Why "not configured" when I *did* configure it

`tdfs4ds` reads `TDFS4DS_*` into `tdfs4ds.<ATTR>` **once, during `import
tdfs4ds`**. A variable exported *after* the import — or a `.env` sitting
outside the process's working directory — never reaches the module config, so
every setting reads as unset. Vector Store auth is the exception: those
variables are read live from `os.environ`, which is why that one check can pass
while the rest report nothing configured.

The **Config load** check names this explicitly rather than letting you hunt for
a setting you already made. The fix is one call:

```python
tdfs4ds.load_config()                              # re-read env + auto-discovered files
tdfs4ds.load_config(dotenv_path='/path/to/.env')   # or point at the file
```

It also flags variables whose value *differs* from the environment — without
failing, since overriding an attribute in-session is legitimate. That catches
the quiet case: an attribute left at its **default** (say
`VECTOR_STORE_BACKEND='chroma'`) while the environment asks for `teradata` is
not "unset", so nothing else notices.

### Exporting a configuration

`tdfs4ds.export_config()` captures a configuration as a file — to reproduce a
session that works, or to turn scattered environment variables into one:

```python
tdfs4ds.export_config()                                  # returns the text; writes nothing
tdfs4ds.export_config('tdfs4ds.json')                    # reusable JSON config
tdfs4ds.export_config('.env', fmt='env', source='env')   # env vars -> a file
```

| Argument | Meaning |
|---|---|
| `source='module'` (default) | Snapshot the **running** configuration, in-session changes included |
| `source='env'` | Convert the `TDFS4DS_*` **environment variables**, whether or not they reached the module config |
| `fmt='json'` | A `tdfs4ds.json` that `load_config()` reads |
| `fmt='env'` | `TDFS4DS_*` lines — the only format that can carry connection and Vector Store auth variables, which are pass-through and never module attributes |
| `include_secrets=False` (default) | Secrets are written as a commented placeholder, so the file is a shareable template |

JSON never carries credentials in any case: `load_config()` rejects them from
JSON by design, so writing them there would leak *and* not work. Both formats
round-trip back through `load_config()`.

## Cloud Provider Setup

The `build_llm()` and `get_embeddings()` functions automatically inject tdfs4ds configuration into environment variables so that LangChain and boto3 clients pick up credentials and endpoints naturally.

### Bedrock (AWS)

```python
import tdfs4ds
from tdfs4ds.agent.consumer_agent import _get_llm
from tdfs4ds.agent.embedding import get_embeddings

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'bedrock'
tdfs4ds.INSTRUCT_MODEL_MODEL = 'anthropic.claude-3-5-sonnet-20241022'
tdfs4ds.INSTRUCT_MODEL_REGION = 'us-east-1'
tdfs4ds.INSTRUCT_MODEL_API_KEY = 'your-bedrock-bearer-token'  # or rely on ~/.aws/credentials

# Auto-injects AWS_BEARER_TOKEN_BEDROCK and AWS_DEFAULT_REGION into the environment
llm = _get_llm()
emb = get_embeddings()
```

AWS credentials are auto-discovered from:
- `AWS_PROFILE` and `~/.aws/credentials`
- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables
- IAM role (when running on EC2, ECS, Lambda, etc.)

### Azure OpenAI

```python
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'azure'
tdfs4ds.INSTRUCT_MODEL_URL = 'https://myinstance.openai.azure.com/'
tdfs4ds.INSTRUCT_MODEL_API_KEY = 'your-azure-api-key'
tdfs4ds.INSTRUCT_MODEL_MODEL = 'gpt-4'
tdfs4ds.INSTRUCT_MODEL_AZURE_DEPLOYMENT = 'my-deployment'  # optional; defaults to model name
tdfs4ds.INSTRUCT_MODEL_AZURE_API_VERSION = '2024-02-01'

# Auto-injects AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY
llm = _get_llm()
emb = get_embeddings()
```

### GCP Vertex AI

```python
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'gcp'
tdfs4ds.INSTRUCT_MODEL_MODEL = 'gemini-1.5-pro-002'
tdfs4ds.INSTRUCT_MODEL_GCP_PROJECT = 'my-gcp-project'
tdfs4ds.INSTRUCT_MODEL_GCP_LOCATION = 'us-central1'

# Auto-injects GOOGLE_CLOUD_PROJECT and CLOUD_ML_REGION
llm = _get_llm()
emb = get_embeddings()
```

GCP authentication via Application Default Credentials:
- Set `GOOGLE_APPLICATION_CREDENTIALS` environment variable to a service account key JSON file
- OR run `gcloud auth application-default login` to use your gcloud credentials
- OR rely on Compute Engine / App Engine / Cloud Functions built-in credentials

### `load_config()` — explicit reload

```python
# Reload from default search paths
tdfs4ds.load_config()

# Point at specific files
tdfs4ds.load_config(
    path='/configs/feature_store.json',
    dotenv_path='/project/.env.production',
)
```

### Priority chain

```
programmatic (tdfs4ds.X = value)
  > OS environment variable (TDFS4DS_X)
  > .env file (./.env or ~/.tdfs4ds/.env)
  > JSON config file (./tdfs4ds.json or ~/.tdfs4ds/config.json)
  > teradataml auto-detection (SCHEMA only)
  > built-in defaults
```

## Time Travel

All catalogs and feature stores are temporal. Point-in-time queries are available via:

```python
tdfs4ds.FEATURE_STORE_TIME = '2024-01-01 00:00:00'   # query historical state
tdfs4ds.FEATURE_STORE_TIME = None                     # back to current state
```

## Cross-Store Read Federation

Connect to your primary store **read-write** as usual, then **extend** the session to one or more **other** feature stores in **read-only** mode. Your uploads still write only to the primary store, but catalog / documentation / lineage **reads** — and `build_dataset` — span every attached store. Each federated row is tagged with a `FEATURE_STORE` provenance column naming its source.

```python
import tdfs4ds

# A — declarative, at connect time (resets the read set to exactly this list):
tdfs4ds.connect(database='DATA_DB',
                databases_readonly=['SHARED_FS', 'TEAM_B_FS'],
                create_if_missing=True)

# B — at runtime, attach / inspect / detach:
tdfs4ds.extend_read_stores('SHARED_FS')                       # a bare schema name
tdfs4ds.extend_read_stores({'schema': 'T_DB',                 # or a table/view-split descriptor
                            'schema_view': 'V_DB'})
tdfs4ds.list_read_stores()                                    # what's attached
tdfs4ds.detach_read_stores('SHARED_FS')                       # or detach_read_stores() for all
```

Also settable via `TDFS4DS_SCHEMA_READONLY` (comma-separated env / `.env`) or the `schema_readonly` JSON key.

- **Byte-identical by default.** With no read-only store attached (`tdfs4ds.SCHEMA_READONLY == []`), every read keeps its exact single-store form — nothing changes until you opt in.
- **Writes never move.** `upload_features`, documentation, and lineage writes always target the primary `SCHEMA`; read-only stores are never modified.
- **Vector search:** with `VECTOR_STORE_BACKEND='teradata'`, semantic search spans read-only stores live (each keeps its own index, no re-index). The local **Chroma** backend searches the primary store only; read-only stores still surface via every catalog / keyword / definition / lineage path.

> **Grants.** The connecting user needs `SELECT` on each read-only store's `FS_V_*` catalog / business-dictionary / lineage views, its `FS_T_*` feature tables (for `build_dataset`), and — with the Teradata vector backend — its vector collections. A store that isn't readable, or is a different tdfs4ds version, is logged and skipped; the others still attach.
>
> **Same-name features.** Resolution is exact when stores hold different data domains / entities / feature names (the usual case). If two stores independently register the same `(data domain, entity, feature name)`, the agent resolves it to the primary store.

## Package Structure

```text
tdfs4ds/
├── __init__.py                    — Global config variables & re-exported public API
├── config.py                      — External config loading (JSON, .env, env vars); load_config()
├── lifecycle.py                   — setup(), connect()
├── execution.py                   — run(), upload_features(), roll_out()
├── catalog.py                     — feature_catalog(), process_catalog(), dataset_catalog()
├── data_domain.py                 — get_data_domains(), select_data_domain(), create_data_domain()
├── datasets.py                    — Utility dataset helpers
├── agent/
│   ├── __init__.py                — Public exports: consumer_agent, query_optimizer, aquery_optimizer, display_optimization_result, …
│   ├── consumer_agent.py          — Intent classifier, 7 skills, feature doc resolver, LLM helpers
│   ├── graph.py                   — LangGraph StateGraph: classify → detect_domain → skill → synthesize
│   ├── query_optimizer.py         — skill_optimize_query(), query_optimizer(), aquery_optimizer(), display_optimization_result()
│   ├── embedding.py               — get_embeddings(), list_embedding_models()
│   ├── vector_index.py            — build_vector_index(), search_vector_index() (Chroma)
│   └── chatbot.py                 — launch_chatbot(), launch_chatbot_with_index() Gradio UI
├── feature_store/
│   ├── entity_management.py       — register_entity(), remove_entity()
│   ├── feature_data_processing.py — prepare_feature_ingestion(), store_feature(), apply_collect_stats()
│   ├── feature_query_retrieval.py — get_list_features(), get_available_features(), get_feature_versions()
│   └── feature_store_management.py — register_features(), feature_store_table_creation()
├── process_store/
│   ├── process_followup.py        — followup_open(), followup_close(), follow_up_report()
│   ├── process_query_administration.py — list_processes(), get_process_id(), remove_process()
│   ├── process_registration_management.py — register_process_view()
│   └── process_store_catalog_management.py — process_store_catalog_creation()
├── dataset/
│   ├── builder.py                 — build_dataset(), build_dataset_opt(), augment_source_with_features()
│   ├── dataset.py                 — Dataset class
│   └── dataset_catalog.py        — DatasetCatalog class
├── genai/
│   └── documentation.py          — LLM-powered auto-documentation of SQL processes (OpenAI / Bedrock)
├── lineage/
│   ├── lineage.py                 — SQL query parsing, DDL analysis
│   ├── network.py                 — Dependency graph construction (on-the-fly)
│   ├── lineage_store.py           — Persisted lineage graph (FS_LINEAGE_GRAPH): rebuild, scoped refresh, impact check, networkx query, pyvis
│   └── indexing.py                — Lineage indexing utilities
└── utils/
    ├── query_management.py        — execute_query(), execute_query_wrapper()
    ├── filter_management.py       — FilterManager class
    ├── time_management.py         — TimeManager class
    ├── lineage.py                 — crystallize_view(), analyze_sql_query(), generate_view_dependency_network()
    ├── info.py                    — update_varchar_length(), get_column_types(), seconds_to_dhms()
    └── visualization.py           — plot_graph(), visualize_graph(), display_table()
```

## GenAI Documentation

The `genai` module provides two complementary ways to document the feature store.

### LLM-powered process documentation

`document_process()` calls an LLM (OpenAI, Azure, vLLM, or AWS Bedrock) to generate:
- Business-logic description of the SQL query
- Entity description and per-column annotations
- EXPLAIN-plan quality metrics: two 1–5 scores (User score / Overall score) plus two deterministic execution counters (`n_steps`, `n_spool_objects`) parsed from the raw Teradata EXPLAIN text, with warnings and recommendations

```python
import tdfs4ds
from tdfs4ds.genai import document_process

# Configure the LLM (or use TDFS4DS_INSTRUCT_MODEL_* env vars / .env file)
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'openai'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'gpt-4o'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = 'sk-...'

process_info = document_process(process_id='<UUID>', show_explain_plan=True)
```

### LLM-powered dataset documentation

`document_dataset_incremental()` documents a **dataset** by walking its full lineage bottom-up:

1. Source tables — uses the business dictionary if available
2. Intermediate views — auto-documented via LLM if undocumented
3. Process views — actively calls `document_process_incremental` if undocumented
4. Feature/entity column descriptions are **propagated** from process docs (no extra LLM call)
5. A single JSON-constrained LLM call generates five structured sections for the dataset

```python
from tdfs4ds.genai import document_dataset_incremental

result = document_dataset_incremental(
    dataset_id   = '<UUID>',  # from dataset_catalog()
    force_update = False,
    upload       = True,
)

# result['DATASET_SECTIONS'] contains:
#   OVERVIEW, ENTITY, FEATURE_THEMES, BUSINESS_QUESTIONS, INTENDED_AUDIENCE
```

Each section is stored as an independent row in `FS_BUSINESS_DICTIONARY_SECTIONS` — no chunking needed for RAG retrieval.

### Full-store documentation in one call

`document_feature_store_incremental()` documents every registered process and dataset in a single optimised pass. Objects are processed in dependency order (leaves first, roots last) so upstream context is always available. A shared pair of visited-sets ensures each process view is documented at most once, even when referenced by multiple datasets.

```python
from tdfs4ds.genai import document_feature_store_incremental

summary = document_feature_store_incremental(
    language     = 'English',
    force_update = False,
    upload       = True,
)
# summary keys: processes_documented, datasets_documented,
#               processes_skipped, datasets_skipped
```

After documentation, process descriptions are automatically mirrored to the business dictionary (object overview + column-level feature descriptions) so the consumer agent can resolve them without any extra step.

## Persisted Lineage Graph

Beyond the on-the-fly `build_teradata_dependency_graph`, tdfs4ds maintains a **persisted, temporal** lineage graph (`FS_LINEAGE_GRAPH`, created when missing at `setup()` / `connect(create_if_missing=True)`). It is updated at every `upload_features` call, so you can cheaply ask **both** "where does this come from?" *and* "what depends on this?" — and drive documentation from it without re-walking DDL.

```python
import tdfs4ds

# Build / fully refresh the graph from every registered process view + dataset.
tdfs4ds.rebuild_lineage_graph()

# Query in-memory (one table read, networkx — falls back to an on-the-fly build
# if the graph was never persisted):
from tdfs4ds.lineage import get_descendants, get_ancestors, get_persisted_graph, plot_lineage_pyvis
get_descendants('my_db', 'vw_aggregate_features')   # impact: what depends on it
get_ancestors('my_db', 'ds_scoring')                # upstream sources it depends on
plot_lineage_pyvis(get_persisted_graph(), output_path='lineage.html')   # interactive HTML

# You edited a (possibly non-process) view? Refresh just its slice and check impact:
tdfs4ds.rebuild_lineage_graph(objects=['vw_aggregate_features'], check=True)
res = tdfs4ds.check_downstream_processes(['vw_aggregate_features'])  # EXPLAIN-compile every consumer
print(res['broken'])   # [{view, process_id, error}, ...] — empty if all still compile

# Re-document from the persisted graph (whole store, or just the impact slice):
tdfs4ds.genai.rebuild_documentation()
tdfs4ds.genai.rebuild_documentation(objects=['vw_aggregate_features'])
```

Object names are case / `"`-quote / `db.name` insensitive and **verified** (`resolve_objects` raises on a typo rather than silently doing nothing). `check_downstream_processes` runs a deterministic `EXPLAIN SELECT *` (compile-only, no execution) on every registered process downstream of a change, so a result-column change that breaks a consumer is caught immediately. The consumer agent's lineage answers now include the downstream / impact direction too.

### Business dictionary (no LLM required)

Three temporal tables store **business-oriented descriptions** for any database object, its columns, and its documentation sections. They form a 3-level hierarchy designed for chunking-free hierarchical RAG:

| Level | Table | Key | Purpose |
|-------|-------|-----|---------|
| 0 | `FS_BUSINESS_DICTIONARY_OBJECTS` | `(DATABASE_NAME, OBJECT_NAME)` | One summary per object (`OBJECT_TYPE`: `'T'`/`'V'`/`'D'`) |
| 1 | `FS_BUSINESS_DICTIONARY_SECTIONS` | `(DATABASE_NAME, OBJECT_NAME, SECTION_NAME)` | One row per documentation section per object |
| 2 | `FS_BUSINESS_DICTIONARY_COLUMNS` | `(DATABASE_NAME, TABLE_NAME, COLUMN_NAME)` | One description per column |

All tables are VALIDTIME temporal and provisioned automatically by `tdfs4ds.connect(create_if_missing=True)`.

```python
import pandas as pd
from tdfs4ds.genai import (
    upload_business_dictionary_objects,
    upload_business_dictionary_columns,
    upload_business_dictionary_sections,
)

# Level 0 — Object-level descriptions
upload_business_dictionary_objects(pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'OBJECT_NAME'         : 'CUSTOMER',
        'OBJECT_TYPE'         : 'T',
        'BUSINESS_DESCRIPTION': 'Core customer table. Each row represents a unique enrolled customer.',
    },
]))

# Level 1 — Section-level descriptions (typically LLM-generated for datasets)
upload_business_dictionary_sections(pd.DataFrame([
    {
        'DATABASE_NAME'  : 'MY_DB',
        'OBJECT_NAME'    : 'DATASET_CUSTOMER',
        'SECTION_NAME'   : 'OVERVIEW',
        'SECTION_CONTENT': 'Customer-level analytical dataset combining spending and category features...',
    },
]))

# Level 2 — Column-level descriptions
upload_business_dictionary_columns(pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'TABLE_NAME'          : 'CUSTOMER',
        'COLUMN_NAME'         : 'CUSTOMER_ID',
        'BUSINESS_DESCRIPTION': 'Unique customer identifier assigned at enrolment.',
    },
]))
```

All three functions validate required columns and perform a `CURRENT VALIDTIME MERGE` — re-running them updates existing descriptions and preserves the full change history.

## Consumer Agent (Chatbot)

The `agent` module provides a conversational interface for business consumers. Non-technical users can ask natural-language questions about features, datasets, definitions, data freshness, usage guidance, data lineage, and calculation logic — in English or French.

### Architecture

```text
User question
  → Intent classifier (Pydantic structured output)
  → DATA_DOMAIN detector (finds which domain owns the feature; remembered across turns)
  → Skill dispatcher (7 skills)
  → Plain-language answer
```

The agent uses LangGraph `StateGraph` with `MemorySaver` for multi-turn conversations. Conversation context is persisted across turns:

| State field | What is remembered |
|---|---|
| `resolved_data_domain` | Which DATA_DOMAIN owns the last named feature/dataset |
| `resolved_object_name` | Last explicitly named feature or dataset |
| `resolved_feature_triplet` | Full resolution: feature name, entity, process ID, view name |
| `resolved_entity_name` | Entity type in focus (e.g. `CustomerID`) |
| `resolved_feature_list` | Feature names currently in focus (one or many) |
| `resolved_column_sources` | Column→source-table map from the last EXPLAIN result (used by DEFINITION drill-down) |

Follow-up questions that omit an explicit feature name (e.g. "when was it last updated?", "how is it calculated?") automatically reuse the previously resolved feature, entity, and domain — no need to repeat yourself. When a feature name is shared across multiple entity types, the remembered entity silently disambiguates without asking for clarification.

After an EXPLAIN turn, the agent lists every **variable involved** in the formula with its source table. Asking "what is `<column>`?" immediately after an EXPLAIN resolves the column through the business dictionary — even if it is not a registered feature. Vague references (e.g. "what does the date mean?") are fuzzy-matched against remembered column and table names.

Feature descriptions are resolved via the process documentation chain:
`entity → features → process_id → VIEW_NAME → FS_BUSINESS_DICTIONARY_COLUMNS`

### Quick start

```python
import tdfs4ds
from tdfs4ds.agent import launch_chatbot_with_index

# Configure LLM and embedding model
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'vllm'
tdfs4ds.INSTRUCT_MODEL_URL      = 'https://api.example.com/v1'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = 'my-key'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'mistral-7b-instruct'
tdfs4ds.EMBEDDING_MODEL_URL     = 'https://api.example.com/v1/e5'
tdfs4ds.EMBEDDING_MODEL_MODEL   = 'bge-m3'

# Build vector index (incremental) then launch the Gradio chatbot — one call
demo = launch_chatbot_with_index(port=7860)
```

Or call the agent programmatically:

```python
from tdfs4ds.agent import consumer_agent

answer = consumer_agent("What features are available?", thread_id="session-1")
answer = consumer_agent("How is nb_days_since_last_transactions calculated?", thread_id="session-1")
answer = consumer_agent("When was it last updated?", thread_id="session-1")  # feature + entity remembered
answer = consumer_agent("What about for CustomerID?", thread_id="session-1")  # entity remembered, new feature group
```

### Skills

The 10 available intents are defined by the `ca-*` SKILL.md files bundled with the
package. Removing a file removes that intent; `reset_consumer_agent()` clears the
singleton cache so the change takes effect without restarting the process.

| Intent | Trigger examples | What happens |
|--------|-----------------|--------------|
| `SEARCH` | "What features analyse customer spending?" | Semantic search across vector index + feature catalog |
| `DEFINITION` | "What does total_amount measure?" | Resolves feature → process view → column doc |
| `USAGE` | "How do I use avg_amount in Tableau?" | Audience, granularity, regulatory guidance |
| `FRESHNESS` | "Is total_amount up to date?" | Checks follow-up execution history |
| `SUMMARY` | "List all available features" | Full feature list with entity and description per feature |
| `LINEAGE` | "Where does total_amount come from?" | Walks upstream dependency graph via `build_teradata_dependency_graph` |
| `EXPLAIN` | "How is total_amount calculated?" | Fetches `SHOW VIEW` DDL → LLM explains logic in plain language + lists source columns so you can drill into any variable |
| `DATASET` | "Which dataset exposes total_amount?" | Looks up dataset catalog for datasets that contain the named feature |
| `DATA_QUERY` | "Show me 20 rows from DS_INVESTIGATION_QUEUE" | Queries registered datasets or base tables (row samples, counts, aggregations); routed to MCP when enabled |
| `BUILD_DATASET` | "Build me a dataset for churn analysis" | Interactive multi-turn flow: resolves entity, proposes features, handles version conflicts, calls `build_dataset()`, auto-documents |

When the user asks about multiple features at once (e.g. "is there a dataset with feature1 **and** feature2?"), the `DATASET` skill returns per-feature results **and** the intersection of datasets that expose all requested features simultaneously.

Restrict the agent to a subset at runtime:

```python
answer = consumer_agent("What features are available?", skills=["SEARCH", "SUMMARY"])
```

Discover installed consumer-agent skills programmatically:

```python
tdfs4ds.consumer_agent_skill_catalog()   # {skill_name: {intent, description, ...}}
```

### MCP Tools (optional)

When `tdfs4ds.MCP_SERVER_URL` is set, the consumer agent can delegate questions to an external [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server — useful for general data queries, external lookups, or calculations that fall outside the feature store domain.

**Configure the endpoint:**

```dotenv
# .env
TDFS4DS_MCP_SERVER_URL=https://your-mcp-server/endpoint/
TDFS4DS_MCP_SERVER_TRANSPORT=streamable_http   # default — modern MCP standard
# TDFS4DS_MCP_SERVER_TRANSPORT=sse             # legacy SSE transport
```

or programmatically:

```python
tdfs4ds.MCP_SERVER_URL       = 'https://your-mcp-server/endpoint/'
tdfs4ds.MCP_SERVER_TRANSPORT = 'streamable_http'   # 'streamable_http' (default) or 'sse'
```

**Editions: Community vs Enterprise**

The two Teradata MCP products differ in how they authenticate and in the tools
they expose, so tell tdfs4ds which one the URL points at:

| Edition | Setting | Credentials |
|---|---|---|
| **Community** (default) | nothing to set | The server embeds its own — tdfs4ds sends no header |
| **Enterprise** | `TDFS4DS_MCP_SERVER_EDITION=enterprise` | HTTP Basic, defaulting to `TDFS4DS_TD_USERNAME` / `TDFS4DS_TD_PASSWORD` |

```dotenv
TDFS4DS_MCP_SERVER_URL=https://your-enterprise-mcp/one-td/mcp
TDFS4DS_MCP_SERVER_EDITION=enterprise
#   -> Basic auth built from the Teradata credentials you already configured

# only if the MCP account differs from the database one:
# TDFS4DS_MCP_USERNAME=...
# TDFS4DS_MCP_PASSWORD=...

# any other scheme (Bearer, API key) — wins over the above:
# TDFS4DS_MCP_SERVER_HEADERS={"Authorization": "Bearer ..."}

# internal certificate authority (verification is never disabled):
# TDFS4DS_MCP_SERVER_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem
```

Credentials are sent **only** when the edition is `enterprise` (or you set
`MCP_AUTH`/`MCP_SERVER_HEADERS` explicitly). That is deliberate: `TD_USERNAME`
and `TD_PASSWORD` exist in nearly every deployment, and an unconditional
fallback would make an existing Community setup start sending the Teradata
password to its endpoint on upgrade. `tdfs4ds.check_configuration()` reports the
edition, the tools it finds, and which variable each credential came from —
never a value.

**Enable MCP in programmatic calls:**

```python
from tdfs4ds.agent import consumer_agent

answer = consumer_agent("How many rows are in the transactions table?",
                        thread_id="session-1", mcp_enabled=True)
```

**Gradio chatbot** — when `MCP_SERVER_URL` is set, the chatbot shows an **Enable MCP Tools** checkbox. Tick it to route applicable questions to the MCP server; leave it unticked for pure feature store mode.

**Safety guardrails** (enforced at both the tool and LLM-prompt level):

| Rule | What happens |
|------|-------------|
| Forbidden objects | SELECT statements against feature store process views or feature storage tables are refused outright |
| Allowed data access | When feature data is needed, only datasets registered in the dataset catalog are queried |
| Row cap | All SELECT queries are automatically rewritten to `SELECT TOP 20`; a ⚠ notice is added to the answer when the limit is reached |

The `langchain-mcp-adapters` package is a lazy optional dependency — it is only imported when MCP is actually invoked. Install it with `pip install langchain-mcp-adapters`.

### Plugin skills

You can extend the agent with custom skills — for example, to run domain-specific analyses or query live data — by pointing `TDFS4DS_SKILLS_FOLDER` at a directory that contains skill subdirectories. Plugin skills are merged transparently into the catalog alongside the bundled `ca-*` skills.

```
$TDFS4DS_SKILLS_FOLDER/
  my-analysis-skill/
    SKILL.md    ← required — defines the intent name and description
    skill.py    ← optional — if present, adds a routable node to the agent graph
```

**`SKILL.md`** uses the same frontmatter format as the bundled skills:

```yaml
---
name: Revenue Trend Analysis
intent: REVENUE_TREND
description: user wants to analyse revenue trends or compare figures across periods
---
```

**`skill.py`** must define a `node(state)` function that receives the full `AgentState` dict and returns a partial state update:

```python
def node(state: dict) -> dict:
    question = state.get("question", "")
    domain   = state.get("resolved_data_domain")
    # ... your analysis logic (teradataml queries, aggregations, etc.) ...
    return {
        "skill_result": {
            "answer": "Revenue grew 12 % YoY, driven by the EMEA region."
        }
    }
```

- If `skill_result` contains an `"answer"` key, it is returned directly to the user without a second LLM pass.
- If `skill.py` is absent, the skill still appears in the intent catalog (its description is included in the classifier prompt) but the agent cannot execute it.
- Plugin skills do **not** need a `ca-` prefix — any directory name under `TDFS4DS_SKILLS_FOLDER` is picked up.

**Setting the folder:**

```python
import os
os.environ["TDFS4DS_SKILLS_FOLDER"] = "/path/to/my/plugins"
```

After changing the folder, call `reset_consumer_agent()` to invalidate the graph cache:

```python
from tdfs4ds.agent import reset_consumer_agent
reset_consumer_agent()
```

### Notebooks

- `09 - Consumer Agent Chatbot with tdfs4ds.ipynb` — architecture walkthrough, 7-intent test suite, 4-turn multi-turn demo
- `10 - Launch Consumer Agent Chatbot.ipynb` — minimal 4-cell one-command launch

### Gradio trace panel

The chatbot includes a collapsible **Agent Trace** accordion showing, for each turn:
- **Intent Classification** — detected intent, object name, domain
- **DATA_DOMAIN Detection** — available domains, resolved domain, source (detected / remembered)
- **Skill Executed** — skill name and inputs
- **Skill Result** — structured output summary; errors include per-step diagnostic messages

### Model listing

```python
from tdfs4ds.agent import list_instruct_models, list_embedding_models

list_instruct_models()                          # models on INSTRUCT_MODEL_URL
list_embedding_models(sub_paths=['e5', 'code']) # models on EMBEDDING_MODEL_URL sub-paths
```

## Query Optimizer Agent

The `query_optimizer` module analyses and rewrites Teradata SQL feature-engineering queries for better performance. It is process-aware — when a `process_id` is supplied it pulls the registered SQL and any stored EXPLAIN documentation directly from the tdfs4ds process catalog, avoiding redundant LLM calls.

### Pipeline (9 steps)

| Step | What happens |
|------|--------------|
| 0 | **Process context** — SQL + stored EXPLAIN analysis fetched from the catalog (skipped when no `process_id`) |
| 0.5 | **SQL simplification** — structural compaction pass merges unnecessary nesting layers into CTE + single outer SELECT, giving the LLM a cleaner baseline; accepts the simplified form only when its EXPLAIN score ≥ original |
| 1 | **Structured EXPLAIN analysis** — `document_sql_query_explain` scores the simplified query 1–5 with `[You]`-prefixed author-actionable warnings and recommendations |
| 2 | **Lineage graph** — Primary Index + partition columns collected for every underlying object |
| 3 | **DDL fetch** — `SHOW TABLE` / `SHOW VIEW` for every referenced object |
| 4 | **Candidate generation** — LLM proposes up to `N` rewrites focused on `[You]`-actionable items (`N = tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATES`, default 5) |
| 5 | **Candidate EXPLAIN** — `document_sql_query_explain` run per candidate |
| 6 | **Plan comparison** — LLM selects the best plan by score delta and resolved warnings |
| 7 | **FilterManager check** — partitioned-but-unfiltered objects are flagged for incremental processing |
| 8 | **Final report** — Markdown with Score Summary, Simplification section, 3-stage query comparison (Input → After Simplification → After Optimization), Candidates, Selected Optimisation, FilterManager |

### Quick start

```python
import tdfs4ds

# Configure LLM — vllm / OpenAI / Azure / Bedrock
tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'vllm'   # 'openai' does not require INSTRUCT_MODEL_URL
tdfs4ds.INSTRUCT_MODEL_MODEL    = '...'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = '...'
# tdfs4ds.INSTRUCT_MODEL_URL = '...'  # required for vllm/azure; omit for openai/bedrock

tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATES       = 5     # max valid rewrites to evaluate (default 5)
tdfs4ds.QUERY_OPTIMIZER_MAX_FAILURES         = 5     # max EXPLAIN/syntax failures before stopping (default 5)
tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATE_TOKENS = None  # cap completion tokens per candidate call (None = model default; set e.g. 4096 for small-context models)

# Process-aware — SQL and stored EXPLAIN docs pulled from the catalog
result = tdfs4ds.query_optimizer(process_id='<UUID>', thread_id='session-1')

# Or pass raw SQL directly
result = tdfs4ds.query_optimizer(
    sql_query="SELECT ... FROM db.tbl",
    thread_id='session-1',
)
```

Multi-turn follow-ups sharing the same `thread_id` use the `MemorySaver` singleton — the agent retrieves the SQL from history and re-runs the pipeline with additional context:

```python
result = tdfs4ds.query_optimizer(
    "Would adding a Secondary Index on the join column improve the plan?",
    thread_id='session-1',
)
```

Inside a Jupyter notebook use the async entry point to avoid background-thread overhead:

```python
from tdfs4ds.agent import aquery_optimizer

result = await aquery_optimizer(process_id='<UUID>', thread_id='session-async')
```

### Result keys

| Key | Content |
|-----|---------|
| `answer` | Structured Markdown optimisation report (Summary, Score Summary, Lineage, …) |
| `best_sql` | Recommended SQL (original if already optimal) |
| `score_delta` | Before/after comparison of scores + execution metrics — see below |
| `original_analysis` | Scored EXPLAIN: `explanation`, `user_score`, `global_score`, `n_steps`, `n_spool_objects`, `warnings`, `recommendations` |
| `candidates` | List of candidate dicts: `sql`, `strategy`, `rationale`, `analysis` |
| `comparison` | Plan comparison: `best_index`, `reasoning`, `business_logic_preserved` |
| `filtermanager_applicable` | `True` if a FilterManager loop was recommended |
| `process_info` | Full process catalog record (when `process_id` is supplied) |
| `steps` | Every pipeline step with inputs and outputs |

### Score comparison (`score_delta`)

After each optimization run, `score_delta` captures exactly how much the rewrite improved the query — across both LLM-assessed scores and deterministic execution-plan metrics:

```python
sd = result['score_delta']
# {
#   'optimized':                True,
#   'best_strategy':            'partition_pruning',
#   'original_user_score':      2,
#   'original_global_score':    3,
#   'best_user_score':          4,
#   'best_global_score':        4,
#   'user_score_delta':         2.0,   # +2 improvement
#   'global_score_delta':       1.0,   # +1 improvement
#   'original_n_steps':         14,
#   'best_n_steps':             9,
#   'steps_delta':              -5,    # 5 fewer execution steps
#   'original_n_spools':        6,
#   'best_n_spools':            4,
#   'spools_delta':             -2,    # 2 fewer spool materialisations
#   'business_logic_preserved': True,
# }
```

Four signals are reported side by side:

- `user_score` (1–5) — quality of what the SQL author controls.
- `global_score` (1–5) — overall plan quality, including infrastructure factors (Primary Index placement, statistics, etc.). A rewrite only improves `user_score`, never `global_score`, if the bottleneck is infrastructure rather than the SQL itself.
- `n_steps` — number of numbered execution steps in the Teradata EXPLAIN plan. Parsed deterministically from the raw EXPLAIN text.
- `n_spool_objects` — number of distinct `Spool` objects the plan materialises. A rough proxy for intermediate-result memory/IO pressure.

Negative `steps_delta` / `spools_delta` mean the rewrite is lighter than the baseline. When the rewrite improves the score but adds a small number of steps or spools (typically single-row CTEs backing a precomputed threshold), the report appends an explanatory note describing the trade-off.

The optimizer `Score Summary` table in the generated report always shows all four metrics as **Before → After** (or **Input → Simplified → Optimized** when the simplification pass changed the SQL).

### Standalone simplification

The simplification pass can be called independently of the full optimizer:

```python
result = tdfs4ds.simplify_query(sql_query="SELECT ...")
# or from a registered process
result = tdfs4ds.simplify_query(process_id='<UUID>')

# result keys: simplified_sql, original_sql, simplified (bool),
#              original_score (1-5), simplified_score (1-5)
if result["simplified"]:
    print(result["simplified_sql"])
```

### Notebook display

`display_optimization_result` renders a score comparison widget at the top of the report, followed by the full Markdown analysis. The widget shows a before/after table with colour-coded Δ Change cells (green for improvement, red for regression) and a footer line for strategy, optimization status, and business-logic preservation.

```python
from tdfs4ds.agent import display_optimization_result

display_optimization_result(result)
```

### FilterManager integration

When `filtermanager_applicable` is `True`, the report includes a ready-to-use code snippet for iterating over partitions one at a time — reducing per-run spool usage and enabling full partition elimination:

```python
fm = tdfs4ds.FilterManager(
    schema_name = tdfs4ds.SCHEMA,
    view_name   = 'TRANSACTIONS',
    col_names   = ['transaction_date'],
)

for filter_id in range(fm.nb_filters):
    fm.update(filter_id)
    tdfs4ds.run(process_id)
```

### Notebook

`11 - Query Optimizer Agent with tdfs4ds.ipynb` (`notebook dev/genai/`) walks through the full pipeline end-to-end — process-aware entry, multi-turn follow-ups, score comparison widget, and FilterManager recommendation.

## HTTP Server (serve)

`pip install tdfs4ds[serve]` adds a production-ready FastAPI server that exposes both agents over an **OpenAI-compatible** API surface — useful for integrating tdfs4ds into existing tools that speak the OpenAI `/v1/chat/completions` protocol.

### API surface

| Route | Method | Description |
|-------|--------|-------------|
| `/v1/models` | GET | Lists available agent model IDs |
| `/v1/chat/completions` | POST | Routes to consumer agent (`tdfs4ds-consumer`) or query-optimizer agent (`tdfs4ds-query-optimizer`) based on the `model` field |
| `/consumer` | GET | LangServe playground for the consumer agent |
| `/query-optimizer` | GET | LangServe playground for the query optimizer |
| `/reports` | GET | List saved query-optimizer report files |
| `/reports/{path}` | GET | Download a specific report (path-traversal-guarded) |

### Running

```bash
# Run directly
python -m tdfs4ds.serve

# Or with Docker
docker build -t tdfs4ds-serve .
docker run --env-file .env -p 8000:8000 -p 7860:7860 tdfs4ds-serve
```

The server also starts a Gradio **admin chatbot** on a second port (default 7860) in the same process, sharing the Teradata context and vector index.

### Environment variables (serve)

Copy `.env.example` to `.env` and fill in the required values:

```dotenv
# --- Teradata connection (required) ---
TDFS4DS_TD_HOST=your-vantage-host
TDFS4DS_TD_USERNAME=your_user
TDFS4DS_TD_PASSWORD=your_password
# TDFS4DS_TD_LOGMECH=LDAP          # optional logon mechanism
# TDFS4DS_TD_DATABASE=your_db      # optional default database for the session
# TDFS4DS_TD_ENCRYPT=true          # optional wire encryption ('true' or 'false')

# --- Feature store scope (required) ---
TDFS4DS_SCHEMA=your_feature_store_db
TDFS4DS_DATA_DOMAIN=your_project

# --- Instruct (LLM) model (required) ---
TDFS4DS_INSTRUCT_MODEL_PROVIDER=openai
TDFS4DS_INSTRUCT_MODEL_MODEL=gpt-4o
TDFS4DS_INSTRUCT_MODEL_API_KEY=sk-...
# TDFS4DS_INSTRUCT_MODEL_URL=https://your-endpoint/v1   # custom / self-hosted

# --- Embedding model (required for consumer agent SEARCH) ---
TDFS4DS_EMBEDDING_MODEL_PROVIDER=openai
TDFS4DS_EMBEDDING_MODEL_MODEL=text-embedding-3-small
TDFS4DS_EMBEDDING_MODEL_API_KEY=sk-...
TDFS4DS_EMBEDDING_MODEL_DIM=1536

# --- Vector store ---
TDFS4DS_VECTOR_STORE_BACKEND=chroma     # 'chroma' or 'teradata'
TDFS4DS_CHROMA_PATH=/data/chroma        # persisted in the /data volume

# --- Server behaviour ---
TDFS4DS_SERVE_PORT=8000
TDFS4DS_SERVE_BUILD_INDEX=true          # set 'false' for a query-optimizer-only deployment
TDFS4DS_SERVE_CHATBOT=true              # launch the Gradio admin chatbot on a second port
TDFS4DS_SERVE_CHATBOT_PORT=7860
```

The full environment-variable table below includes all `TDFS4DS_TD_*` and `TDFS4DS_SERVE_*` variables.

### Per-user isolation

The serve module is process-wide: all requests share one Teradata context and one Chroma collection. For strict per-user isolation (separate `DATA_DOMAIN`, separate vector index), deploy one container per user with its own `TDFS4DS_DATA_DOMAIN` and `TDFS4DS_CHROMA_PATH` / volume. A `docker-compose.yml` template is included in the repository root.

## Discover Registered Features

```python
from tdfs4ds.feature_store.feature_query_retrieval import (
    get_list_entity,
    get_list_features,
    get_available_features,
    get_feature_versions,
)
```

## Lineage

The `lineage` module builds end-to-end dependency graphs from a SQL query or a dataset view DDL.

### Dependency graph

```python
from tdfs4ds.lineage import build_teradata_dependency_graph, plot_lineage_sankey, show_plotly_robust

# Start from a dataset view DDL (obtained via SHOW VIEW)
sql = tdml.execute_sql("SHOW VIEW DATASET_CUSTOMER").fetchall()[0][0]

graph = build_teradata_dependency_graph(sql_query=sql)
# Returns: {"nodes": {...}, "edges": [...], "roots": [...]}
```

By default (`expand_datasets_via_process_catalog=True`) dataset nodes are resolved through
the process catalog: `FEATURE_VERSION` UUIDs embedded in the dataset DDL are matched to
`PROCESS_ID` entries in `FS_V_PROCESS_CATALOG`, and edges are drawn directly to the
registered feature-engineering views.

```
DATASET_CUSTOMER  →  FEAT_ENG_CUST  →  DB_SOURCE.TRANSACTIONS
```

Set `expand_datasets_via_process_catalog=False` to connect the dataset directly to the
raw feature-store storage tables (previous behaviour).

```python
fig = plot_lineage_sankey(graph, title="Customer Dataset Lineage")
show_plotly_robust(fig)
```

### Migration manifest

`graph_to_migration_manifest` converts any lineage graph into a flat, JSON-serialisable
dict — useful for planning a feature store migration.

```python
from tdfs4ds.lineage import graph_to_migration_manifest
import json

# All databases
manifest = graph_to_migration_manifest(graph)

# Scoped to the feature store schema only (cross-boundary edges excluded)
manifest_fs = graph_to_migration_manifest(graph, filter_database=tdfs4ds.SCHEMA)
print(json.dumps(manifest_fs, indent=2))
# {
#   "views":  [{"database": "demo_user", "name": "DATASET_CUSTOMER", "type": "dataset"},
#              {"database": "demo_user", "name": "FEAT_ENG_CUST",    "type": "view"}],
#   "tables": [],
#   "edges":  [{"from": "demo_user.DATASET_CUSTOMER", "to": "demo_user.FEAT_ENG_CUST"}]
# }

with open("migration_manifest.json", "w") as f:
    json.dump(manifest_fs, f, indent=2)
```

## Claude Code Skills

`tdfs4ds` ships **25 bundled `SKILL.md` files** organized in five families that teach
Claude Code (and compatible agents) how to drive the feature store end-to-end — from
first connection to consumer chatbot. This includes:
- 10 **workflow skills** (`fs-*`) — core feature store operations
- 2 **Teradata utilities** (`td-*`) — EXPLAIN visualization and end-to-end analytics workflow
- 3 **query optimizer skills** (`qo-skill-*`) with 11 nested LLM prompts
- 10 **consumer-agent reference skills** (`ca-*`) that document each intent of the conversational agent

### Installing after `pip install tdfs4ds`

**Convenience functions** — recommended for most users:

```python
import tdfs4ds

# Install globally (all projects on this machine)
tdfs4ds.install_skills_global()

# Or install locally (just this project — put .claude/skills/ in git)
tdfs4ds.install_skills_local()

# Install a specific subset of skills
tdfs4ds.install_skills_global(skills=['fs-setup', 'fs-upload', 'ca-search'])
tdfs4ds.install_skills_local(skills=['fs-document', 'fs-lineage', 'fs-analyze'])
```

**Lower-level function** — when you need custom target directories:

```python
# Copy all skills to a custom location (e.g. ~/.config/mycli/skills/)
tdfs4ds.install_skills(target_dir='~/.config/mycli/skills')

# Only add skills that do not exist yet (safe for shared project dirs)
tdfs4ds.export_skills('.claude/skills')        # overwrite=False by default

# Install a specific subset into a custom location
tdfs4ds.install_skills(target_dir='./backup/skills', skills=['fs-setup', 'fs-upload'])
```

Or from the **command line** with the venv active:

```bash
# User-level (all projects on this machine)
tdfs4ds-install-skills

# Project-level
tdfs4ds-install-skills --target .claude/skills

# Skip skills already present
tdfs4ds-install-skills --target .claude/skills --no-overwrite

# List bundled skill names
tdfs4ds-install-skills --list
```

### Skill catalogue

**Feature Store Workflow Skills** (`fs-*`)

| Skill | Purpose |
|-------|---------|
| `fs-setup` | Connect to Teradata, run `setup()`, activate a data domain |
| `fs-upload` | Engineer features in SQL / teradataml, register with `upload_features` |
| `fs-tdstone2` | Register a trained tdstone2 model's per-partition outputs as governed features |
| `fs-filter` | Segmented ingestion with `FilterManager` (standard, hybrid, clone) |
| `fs-rollout` | Backfill across a date range with `TimeManager` + `roll_out` |
| `fs-dataset` | Resolve feature versions, build a denormalised dataset view |
| `fs-inspect` | Browse process / feature / dataset catalogs and follow-up table |
| `fs-document` | LLM process documentation + EXPLAIN score 1–5 |
| `fs-lineage` | Build a dependency graph and render a Sankey diagram |
| `fs-analyze` | Scalability analysis — EXPLAIN + partition opportunities + FilterManager proposal |
| `fs-agent` | Launch the consumer agent chatbot (LangGraph + Chroma + Gradio) |

**Teradata Utilities** (`td-*`)

| Skill | Purpose |
|-------|---------|
| `td-explain` | Interactive HTML flow diagram from any raw Teradata EXPLAIN output |
| `td-analytics-workflow` | End-to-end analytics pipeline: rule analytics → feature engineering → in-database ML (tdstone2) → explainability → agentic consumption |

**Query Optimization Skills** (`qo-skill-*` with nested prompts)

| Skill | Purpose | Nested Prompts |
|-------|---------|---|
| `qo-skill-explain` | Analyze EXPLAIN plans and score query quality (1–5) | 3 prompts: EXPLAIN analysis, false-positive classification, SQL documentation |
| `qo-skill-simplify` | Flatten SQL nesting and remove redundant wrappers | 2 prompts: simplification logic, syntax repair loop |
| `qo-skill-optimize` | Full optimization pipeline: EXPLAIN → strategies → candidates → best plan | 6 prompts: strategies, generation, refinement, candidate comparison, summary, fallback generation |

**Consumer-agent reference skills** (also used by `build_graph()` to configure the agent at runtime)

| Skill | Intent | Trigger example |
|-------|--------|-----------------|
| `ca-search` | `SEARCH` | "What features measure customer spending?" |
| `ca-definition` | `DEFINITION` | "What is `total_amount`?" |
| `ca-usage` | `USAGE` | "How do I use `avg_amount` in Tableau?" |
| `ca-freshness` | `FRESHNESS` | "Is `total_amount` up to date?" |
| `ca-summary` | `SUMMARY` | "Give me a snapshot of the feature store" |
| `ca-lineage` | `LINEAGE` | "Where does `total_amount` come from?" |
| `ca-explain` | `EXPLAIN` | "How is `total_amount` calculated?" |
| `ca-dataset` | `DATASET` | "Which dataset exposes `total_amount`?" |
| `ca-data-query` | `DATA_QUERY` | "Show me 20 rows from DS_FRAUD_DETECTION" |
| `ca-build-dataset` | `BUILD_DATASET` | "Build me a dataset for churn analysis" |

### Sharing skills with teammates

Choose between **global** (user-level) and **local** (project-level) installation based on your workflow:

**Global installation** — shared across all projects on this machine:
```python
tdfs4ds.install_skills_global()
# Skills available in ~/.claude/skills/ for Claude Code, IDE extensions, and CLI
```

Use this when:
- You work solo or each team member manages their own Claude Code environment
- You want one-time setup and skills available everywhere
- You don't need to version-control the skills with the project

**Local installation** — one-per-project, version-controlled in git:
```python
tdfs4ds.install_skills_local()
# Skills available in .claude/skills/ (commit to git for team sharing)
```

Use this when:
- You have a shared git repository and want teammates to get skills automatically on `git pull`
- Your team has customized skills (edited SKILL.md files) that should be tracked
- You want skills pinned to a specific package version

| Function | Target | Scope | Commit to git? |
|----------|--------|-------|----------------|
| `install_skills_global()` | `~/.claude/skills/` | All projects for this user | No — personal setup |
| `install_skills_local()` | `.claude/skills/` | This project only | Yes — shared with team |

## Requirements

- Python >= 3.6
- teradataml >= 17.20
- Active Teradata Vantage connection
- **VALIDTIME temporal tables must be enabled** on the Teradata Vantage system — all feature catalogs, process catalogs, and feature stores rely on `VALIDTIME` support
