Metadata-Version: 2.4
Name: prizm-adls-scanner
Version: 0.1.2
Summary: Prizm Databricks Exception, Lookup, Comparison, and Materialize job runtime
Author-email: Prizm Team <team@prizm.com>
Maintainer-email: Prizm Team <team@prizm.com>
License: MIT
Project-URL: Homepage, https://github.com/DQLabs-Inc/prizm-cli
Project-URL: Repository, https://github.com/DQLabs-Inc/prizm-cli.git
Project-URL: Bug Tracker, https://github.com/DQLabs-Inc/prizm-cli/issues
Keywords: prizm,databricks,exception,lookup,comparison,data-quality
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: sqlglot>=20.0.0
Provides-Extra: dev
Requires-Dist: pre-commit==2.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov==2.0; extra == "test"
Requires-Dist: pytest-mock==3.0; extra == "test"
Dynamic: license-file

# prizm-adls-scanner

[![Python Version](https://img.shields.io/badge/python-3.11+-blue.svg)](https://python.org)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Databricks](https://img.shields.io/badge/Databricks-Supported-red.svg)](https://databricks.com)

Python package that runs **Prizm Exception, Lookup, Comparison, Materialize, and Behavioral jobs** inside a Databricks Spark cluster.

**Location:** `prizm-cli/prizm-adls-scanner` (under Server root)

**What it is:**
- ✅ A library imported **inside a Databricks notebook or job task** — it does not read files or connect to storage itself; it operates on an already-active `spark` session
- ✅ Runtime deps are just `requests` and `sqlglot` — `pyspark` is intentionally **not** a dependency (Databricks already provides Spark on the cluster; installing PyPI `pyspark` there would conflict with it)
- ✅ `pyspark` is imported lazily inside each runner module, so `pip install`/`import prizm_adls_scanner` succeeds even outside Databricks (useful for local dev of non-Spark code paths)

**What it is not:**
- ❌ Not a CLI — no console script, no `argparse`/`click` entry point
- ❌ Not a file scanner/connector — it has no ADLS/S3 I/O logic; it assumes data is already loaded into Spark by the notebook/job that calls it
- ❌ Not the metrics-collection scanner — that's a different, unrelated concern

---

## Install

### Databricks notebook

```python
%pip install prizm-adls-scanner
dbutils.library.restartPython()
```

### Cluster / job library (recommended for production)

Attach the published wheel as a cluster or job library so tasks don't need `%pip` on every run.

### From a built wheel

```bash
make build
pip install dist/prizm_adls_scanner-*.whl
```

**Note:** `pyspark` is **not** a package dependency. Databricks already provides Spark. Do not install PyPI `pyspark` on the cluster.

---

## Databricks notebook usage

Use a thin notebook; all job logic lives in the package:

```python
%pip install prizm-adls-scanner
dbutils.library.restartPython()

import json
from prizm_adls_scanner.databricks import run_metric_notebook_job

WIDGET_NAMES = [
    "job_config",
    "storage_account",
    "access_key",
    "client_id",
    "client_secret",
    "tenant_id",
    "uc_catalog",
    "uc_schema",
    "exception_url",
    "run_id",
    "file_batch",
    "job_type",
    "connector",
    "container",
    "asset_id",
    "source_id",
]

for name in WIDGET_NAMES:
    try:
        dbutils.widgets.text(name, "")
    except Exception:
        pass

widgets = {}
for name in WIDGET_NAMES:
    try:
        widgets[name] = dbutils.widgets.get(name)
    except Exception:
        widgets[name] = ""

result = run_metric_notebook_job(
    spark=spark,
    dbutils=dbutils,
    widgets=widgets,
)
dbutils.notebook.exit(json.dumps(result, default=str))
```

A reference notebook source also ships in the repo as `prizm_adls_scanner/databricks/notebook/prizm_iceberg_notebook.py`.

---

## API

```python
from prizm_adls_scanner.databricks import (
    run_metric_notebook_job,   # routes by widgets["job_type"] to the right runner
    run_exception_job,
    run_exception_notebook_job,
    run_lookup_job,
    run_comparison_job,
    run_materialize_job,
)
# also available directly off the top-level package (lazy import):
from prizm_adls_scanner import run_exception_job
```

| Argument | Description |
|---|---|
| `spark` | Active Databricks `SparkSession` |
| `dbutils` | Optional `dbutils` |
| `widgets` | Dict of notebook/job widget values |

Every runner returns a `dict` with at least `status` (`success` / `failed`). On failure, includes `error` / `errors`.

`run_metric_notebook_job` routes by `widgets["job_type"]`:
- `EXCEPTION` / `EXCEPTION_METADATA` → exception runner
- `LOOKUP` → lookup runner
- `COMPARISON` → comparison runner
- `BEHAVIORAL` → side-snapshot runner
- `MATERIALIZE_ICEBERG` / `DELETE` (or `is_delete_action`, or no `job_type`) → materialize runner (ADLS → Unity Catalog Delta UniForm)

---

## Development

### Development Setup

```bash
pip install -e ".[dev,test]"
```

### Running Tests

Most of the test suite runs without `pyspark` installed (package-surface/import checks); a couple of tests are skipped automatically if `pyspark` isn't present.

```bash
make test
```

### Building Wheels

```bash
make build
```

For offline/local environments where build dependencies are already installed:

```bash
make build-local
```

### Clean Build Artifacts

```bash
make clean
```

---

## Package Layout

```text
prizm_adls_scanner/
  __init__.py                     # lazy top-level re-exports
  databricks/
    __init__.py                   # run_exception_job, run_lookup_job, run_comparison_job,
                                   # run_materialize_job, run_metric_notebook_job
    _runtime.py                   # shared runtime helpers
    exception_runner.py           # EXCEPTION / EXCEPTION_METADATA runtime
    exception_notebook.py         # reference notebook source
    exception_metadata_runner.py  # exception metadata runtime
    lookup_driver.py / lookup_runner.py         # LOOKUP runtime
    comparison_driver.py / comparison_runner.py # COMPARISON runtime
    materialize_runner.py         # MATERIALIZE_ICEBERG / DELETE runtime
    side_snapshot_runner.py       # BEHAVIORAL runtime
    notebook/prizm_iceberg_notebook.py  # reference Iceberg notebook source
```

---

## License

MIT License

## Support

For issues and questions, please visit:
https://github.com/DQLabs-Inc/prizm-cli/issues
