Metadata-Version: 2.4
Name: det-elt
Version: 0.5.0
Summary: DET — Data Extract Tool: config-driven extraction runtime with pluggable ingestion
Author-email: Kadeem <kletts2000@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/codekid/det
Project-URL: Repository, https://github.com/codekid/det
Project-URL: Issues, https://github.com/codekid/det/issues
Project-URL: Documentation, https://github.com/codekid/det/blob/main/docs/getting-started-library.md
Keywords: etl,data-extraction,pipeline,iceberg,bronze
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: <3.14,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jsonschema>=4.23.0
Requires-Dist: pendulum>=3.0.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: requests>=2.32.0
Requires-Dist: structlog>=25.1.0
Requires-Dist: typer>=0.15.0
Provides-Extra: dev
Requires-Dist: pytest>=8.3.0; extra == "dev"
Requires-Dist: pytest-cov>=6.0.0; extra == "dev"
Requires-Dist: ruff>=0.9.0; extra == "dev"
Provides-Extra: examples
Requires-Dist: beautifulsoup4>=4.12.0; extra == "examples"
Requires-Dist: dlt[filesystem]>=1.24.0; extra == "examples"
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.5.1; extra == "duckdb"
Provides-Extra: scaffold
Requires-Dist: jinja2>=3.1.0; extra == "scaffold"
Provides-Extra: dbt
Requires-Dist: dbt-core>=1.9; extra == "dbt"
Requires-Dist: dbt-duckdb>=1.9; extra == "dbt"
Provides-Extra: bigquery
Requires-Dist: dbt-bigquery>=1.9; extra == "bigquery"
Requires-Dist: google-cloud-bigquery>=3.0; extra == "bigquery"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
Provides-Extra: s3
Requires-Dist: s3fs>=2024.0.0; extra == "s3"
Provides-Extra: gcs
Requires-Dist: gcsfs>=2024.0.0; extra == "gcs"
Provides-Extra: iceberg
Requires-Dist: pyiceberg>=0.8; extra == "iceberg"
Requires-Dist: pyarrow>=15; extra == "iceberg"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.2; extra == "mcp"
Dynamic: license-file

# DET — Data Extract Tool

Extract → **raw** (wire bytes) → **bronze** (typed Iceberg). **dbt** owns silver and gold.
**dlt never lands bronze** — it may help HTTP; DET owns validation, meta, and writers.

## Choose your path

| Audience | Start here |
| --- | --- |
| **Operator** (this monorepo: CLI, examples, dbt, Airflow, MCP) | [Try it](#try-it-fixtures-no-noaa-download) below |
| **Library** (embed extract→raw→bronze in your app) | **[docs/getting-started-library.md](docs/getting-started-library.md)** · [docs/api.md](docs/api.md) |

This checkout is the operator product **and** the library source. Embedders install
the PyPI package **`det-elt`** (import `det`) and keep pipelines under
*their* `project_root` — they do not need this monorepo’s dbt/Airflow layout.

---

## Try it (fixtures, no NOAA download)

Python 3.12+ and [uv](https://github.com/astral-sh/uv):

```bash
uv venv && make install          # recommended extras (includes iceberg + examples)
export DET_LAKE_PATH="$PWD/data/lake"

uv run det run -p noaa.storm_events -s 2026-08-06 \
  --set source.overrides.local_csv_dir=fixtures/storm_events \
  --set source.overrides.filename_substr=details

uv run det dbt -p noaa.storm_events
duckdb data/analytics.duckdb -c "select * from gold.gold_yearly_damage"
```

You should see:

- Raw: `data/lake/raw/noaa/storm_events_v1/…/data/` + `meta/manifest.json`
- Bronze: Iceberg table at `data/lake/bronze/noaa/storm_events_v1/`
- Silver/gold in `data/analytics.duckdb`

`make run-local` is a **thin JSONL** smoke (`destination.type=filesystem`). It does not
feed Iceberg `iceberg_scan` in dbt. Use the commands above for the full path, or
`make all` only if you intend JSONL bronze.

Use `uv run det …` (or `make …`). If macOS hides the editable `.pth`, `make unhide` or
`PYTHONPATH=src .venv/bin/det …`.

---

## Mental model

```mermaid
flowchart LR
  src[Source plugin] --> extract
  extract --> raw[Raw lake]
  raw --> load
  load --> bronze[Iceberg bronze]
  bronze --> stg[dbt stg]
  stg --> silver --> gold
```

| Layer | Owner | Rule |
| --- | --- | --- |
| Raw | DET | Append extract runs; rebuild source of truth |
| Bronze | DET | Typed landing; prune bronze only, never raw |
| Silver / gold | dbt | Dedupe and marts; latest extract wins |

**Interval:** `-s` inclusive start, `-e` exclusive end (default start + 1 day).
`det run` shares one `__extract_run_datetime` across raw, bronze, and rows.

---

## Install extras

Base `det` is the runtime + CLI (no Iceberg / DuckDB / example HTTP deps).
**Recommended first line:** install with the `iceberg` extra.

| Extra | For |
| --- | --- |
| `iceberg` | **Recommended** — default lake bronze |
| `examples` | In-tree HTTP sources (`dlt`, BeautifulSoup) |
| `duckdb` | DuckDB destination / DuckDB-backed prune |
| `scaffold` | Jinja2 dbt/pipeline scaffolding |
| `dbt` | `det dbt` |
| `postgres` | SQL serving destination |
| `s3` / `gcs` | Object-store lake URI |
| `mcp` | Cursor inspect / dry-run server |
| `dev` | pytest, ruff |

```bash
# App / embed (PyPI):
uv pip install "det-elt[iceberg]"
uv pip install "det-elt[iceberg,duckdb,dbt]"
# This checkout (editable):
uv pip install -e ".[iceberg]"              # recommended
uv pip install -e ".[iceberg,duckdb,dbt]"   # typical app repo
make install                                # full extras for operator dev
make test
uv run det check
```

---

## Pipeline YAML

Canonical id is `provider.source` (`-p noaa.storm_events`). Defaults live in the
plugin; YAML wires schema, destination, and optional dbt knobs. Plugins are
discovered from `src/det/sources/<provider>/<source>.py` (`name` must equal
`provider.source`) — they are not listed in `plugins.py`. Out-of-tree packages may
use entry points `det.sources` / `det.mappers`.

```yaml
name: noaa.storm_events
source:
  type: noaa.storm_events
destination:
  type: iceberg          # default lake; filesystem = JSONL; duckdb / postgres = SQL
  partition: extract_run # Iceberg only; omit = extract_run; small tables: none
wire_version: 1          # lake id is always {name}_vN (including _v1)
```

Schema defaults to `schemas/<provider>/<source>/<source>.schema.yaml`.
Greenfield: `det init-pipeline --name example_api.events --source-type example_api.events`.

---

## Destinations

Lake root: `DET_LAKE_PATH` / `--lake-path` (default `./data/lake`). Same hive under
one root (`raw/` + `bronze/` prefixes) for layout 1. Optional **layout 2** split
roots: `DET_LAKE_PATH_RAW` / `_BRONZE` / `_OPS` (or `DetSettings.lake_path_*`) —
arbitrary bucket URIs you choose; flattened `{provider}/{source}_vN` under each.
There is no `destination.type: s3`. See [docs/lake-layout.md](docs/lake-layout.md).

**`DET_LAKE_MODE`** (policy around the URI; unset → `local`):

| Mode | Allowed lake | Typical use |
| --- | --- | --- |
| `local` | filesystem path or `memory://` (tests) | laptop, CI default suite, Compose default |
| `cloud` | `s3://…` or `gs://…` / `gcs://…` | object store; CI MinIO soak covers S3 extract→Iceberg→`iceberg_scan`/`det dbt`; GCS soak covers extract→Iceberg (PyIceberg) |

`--lake-path` cannot bypass mode. `det check` errors on mismatch and warns when
`mode=cloud`. Compose: `DET_LAKE_MODE` + overridable `DET_LAKE_PATH` (see
`airflow/.env.example`). Local analytics/ops DuckDB stay on the worker filesystem.
Prod on GCS: BigQuery reads Iceberg bronze via **BigLake** — see
[docs/gcp-biglake.md](docs/gcp-biglake.md) (architecture C). For multi-engine
metastores (`rest` / `glue`), see [docs/iceberg-catalog.md](docs/iceberg-catalog.md)
(`DET_ICEBERG_CATALOG`, default `hadoop`). Local Polaris+MinIO: `make polaris-up`.
Publish existing Hadoop tables with `det iceberg-register --dry-run` then
`--apply` (exactly one required) after switching catalog — see
[docs/iceberg-catalog.md](docs/iceberg-catalog.md).

| `destination.type` | Bronze |
| --- | --- |
| **`iceberg`** | Default. Parquet table at `<lake>/bronze/<provider>/<source>_vN/` |
| `filesystem` | Hive JSONL (thin / fixtures). Cannot share that path with Iceberg |
| `duckdb` | `bronze_{provider}.{source}_vN` — needs `connection` |
| `postgres` | Same SQL names — `connection_env: DET_POSTGRES_DSN` (never a DSN in YAML) |

There is **no** `destination.type: bigquery`. On `gs://`, bronze stays Iceberg;
BQ is a reader (BigLake), not a DET lander.

Iceberg-only: `destination.partition` is `extract_run` (default — identity on
`__extract_run_datetime` for load replace / silver watermark) or `none`
(unpartitioned; use for small tables). Spec applies on **create** only. Live
mismatch **hard-fails** until `det migrate … --recreate-iceberg` (full table
purge, then rewrite `-s`/`-e` or `--all-raw`; latest raw per interval unless
`--all-raw-runs`) or a manual wipe. Not the raw hive layout.

`det dbt -p …` sets `DET_BRONZE_SOURCE` from the pipeline (`iceberg` → `iceberg_scan`
in `sources.yml` for DuckDB). On `s3://` lakes, `det dbt` auto-selects profile
target `duckdb_s3` (httpfs + S3 secret from the same `AWS_*` as extract/load).
On `gs://`, it does **not** force DuckDB S3 — set `DET_DBT_TARGET=bigquery` /
`--target bigquery` for BigLake-backed silver/gold/ops ([docs/gcp-biglake.md](docs/gcp-biglake.md)).
Register BigLake tables with `det biglake-register` (dry-run → `det approve` → `--apply`).
See [docs/gcp-biglake.md](docs/gcp-biglake.md) for connection IAM and sandbox teardown.
Layout contract: [docs/lake-layout.md](docs/lake-layout.md).

---

## Everyday CLI

`-p` is a canonical id, `provider/source`, or a YAML path. Project root:
`--project-root` > `DET_PROJECT_ROOT` > cwd.

```bash
det run -p noaa.storm_events -s 2026-08-06
det extract -p noaa.storm_events -s 2026-08-06
det load -p noaa.storm_events -s 2026-08-06
det dbt -p noaa.storm_events
det check          # CI runs det check --strict (warnings fail the build)
det list-pipelines
det list-sources
```

| Command | When |
| --- | --- |
| `det prune -p … -s … --keep 1 --dry-run` then `--apply` | Drop old bronze extract-run siblings (never raw) |
| `det migrate -p … --to-bronze … --schema … --mapper identity -s … -e …` | Rebuild bronze from raw after a contract change |
| `det scaffold-dbt -p …` | Emit stg/silver + `sources.yml` |
| `det scaffold-ops` | Emit ops dbt models/tests/macros + SLO seed (after `runs-materialize`) |
| `det runs` / `det runs-materialize` | Attempt receipts; optional ops Iceberg + `det dbt --select tag:ops` |
| `det lock-show` / `lock-release --force` | Lake lease on `(pipeline, interval)` if a worker died |

Logs: console on a laptop TTY, JSON off TTY (`DET_LOG_FORMAT`). Secrets: names in
YAML (`auth_env`, `connection_env`); values in env. `det check` fails passwordful DSNs
in committed config.

---

## dbt

```bash
det dbt -p noaa.storm_events    # stg_<provider>__<source>+ ; sets lake env
det dbt                         # full analytics project (excludes tag:ops)
```

Scaffolded `stg_*` use `det_bronze_from`. Gold is hand-written. Nested flatten /
relations: `dbt.stg` in pipeline YAML, then `det scaffold-dbt -p …`. Contract
triangle (schema → sources.yml → dbt.stg): [docs/contract-triangle.md](docs/contract-triangle.md).
More: [`.cursor/skills/det-dbt/SKILL.md`](.cursor/skills/det-dbt/SKILL.md).

---

## Local Airflow (dev only)

```bash
make airflow-up          # http://localhost:8080  — airflow / airflow
make airflow-down
```

Compose is **LocalExecutor**, not production. DAGs: extract → load, silver/gold dbt,
ops receipts — decoupled. Env: `airflow/.env.example`. Guide:
[`.cursor/skills/det-airflow/SKILL.md`](.cursor/skills/det-airflow/SKILL.md).

---

## Cube (local semantic layer)

Gold and ops metrics for agents. Not Cube Cloud MCP.

```bash
make cube-up             # http://localhost:4000
make cube-down
```

Compose reads `data/analytics.duckdb` (`yearly_damage`) and `data/det_ops.duckdb`
(`run_daily`). Copy `cube/.env.example` → `cube/.env`. MCP: `cube_meta` /
`cube_load` (`DET_CUBE_BASE_URL`, `DET_CUBE_API_SECRET`). Do not run `det dbt`
against a DuckDB file Cube has open. Certified metrics go through Cube; silver
or ops row detail uses MCP `query_analytics`.

---

## MCP (Cursor)

Read-only inspect + dry-run (no extract/load/prune-apply/DagRuns).
`uv pip install -e ".[mcp]"`; [`.cursor/mcp.json`](.cursor/mcp.json) already launches it.
Agent contract: [AGENTS.md](AGENTS.md). Tools + policy:
[`.cursor/rules/det-mcp.mdc`](.cursor/rules/det-mcp.mdc).

---

## Repo map

```text
src/det/                 CLI, runtime, example sources, writers, det.testing
configs/pipelines/       provider.source YAML
schemas/                 bronze JSON Schema
docs/api.md              public Python API (SemVer / __all__)
docs/getting-started-library.md  embedder first hour
docs/lake-layout.md      hive / SQL compatibility
docs/gcp-biglake.md      gs:// Iceberg + BigLake + dbt-BQ (architecture C)
dbt/                     silver + gold + ops
cube/                    local Cube Core (gold + ops metrics)
dags/ + airflow/         local Compose
fixtures/                offline NOAA CSVs
```

Example sources implement `extract_to_raw` / `records_from_raw`. Land near-wire bytes;
unexpected fields fail JSON Schema. Analytics renames live in `dbt.stg`. True wire
breaks bump `wire_version`. dlt: `RESTClient` / `@dlt.resource` as iterators only —
never `dlt.pipeline` for landing.

---

## Troubleshooting

| Symptom | Fix |
| --- | --- |
| `No module named 'det'` | `make unhide` or `PYTHONPATH=src .venv/bin/det` (macOS hidden `.pth`) |
| `dbt CLI not found` | `uv run det dbt` so the venv `dbt` is on PATH |
| `No raw partitions` | Load the **same** `-s`/`-e` you extracted |
| Iceberg partition YAML ≠ live | `det migrate … --recreate-iceberg` (or wipe the bronze table path); plain load/migrate hard-fails |
| Iceberg dbt vs JSONL lake | Don’t mix `make run-local` (filesystem) with `det dbt -p` (pipeline is iceberg) |
| DuckDB lock in Airflow | Absolute `DET_ANALYTICS_DUCKDB`; one dbt process at a time |
| Cube MCP `cube_unavailable` | `make cube-up`; copy `cube/.env.example` → `cube/.env` |
| DuckDB lock with Cube | Do not `det dbt` while Cube has that file open |
| `DET_LAKE_MODE=local forbids…` / `requires an s3://` | Align mode and path: local + filesystem, or cloud + `s3://`/`gs://` |
| GCS soak / localgcp | Set `STORAGE_EMULATOR_HOST` (`http://127.0.0.1:4443`), `GOOGLE_CLOUD_PROJECT`, `DET_GCS_BUCKET`; `pytest -m gcs`. Lakehouse REST on real GCP: [docs/gcp-lakehouse-soak.md](docs/gcp-lakehouse-soak.md) (`pytest -m lakehouse`). BigLake BQ externals: [docs/gcp-biglake.md](docs/gcp-biglake.md) |
