Metadata-Version: 2.5
Name: airflow-lineage-toolkit
Version: 1.0.0
Summary: Static, runtime-free lineage extraction, querying and visualization for Apache Airflow DAGs.
Project-URL: Homepage, https://github.com/pavankalyan03/airflow-lineage-toolkit
Project-URL: Repository, https://github.com/pavankalyan03/airflow-lineage-toolkit
Project-URL: Issues, https://github.com/pavankalyan03/airflow-lineage-toolkit/issues
Project-URL: Changelog, https://github.com/pavankalyan03/airflow-lineage-toolkit/blob/main/CHANGELOG.md
Author-email: Pavan Kalyan Kotha <pavankalyan.kotha.03@gmail.com>
Maintainer-email: Pavan Kalyan Kotha <pavankalyan.kotha.03@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Pavan Kalyan Kotha
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: airflow,dag,data-engineering,impact-analysis,lineage,static-analysis
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == 'excel'
Provides-Extra: ui
Requires-Dist: fastapi>=0.100; extra == 'ui'
Requires-Dist: uvicorn>=0.23; extra == 'ui'
Description-Content-Type: text/markdown

# airflow-lineage-toolkit

**Answer "what breaks if I change this DAG?" without running Airflow.**

`airflow-lineage-toolkit` reads your Airflow DAG *source files* with Python's `ast`
module and builds a complete dependency graph: every task, every `>>`, every
cross-DAG sensor and trigger, plus what each task actually runs (Databricks main
class / JAR / notebook / wheel, dbt Cloud job id).

No Airflow instance. No metadata database. No importing your DAG code. Just the
standard library pointed at a folder.

```bash
pip install airflow-lineage-toolkit

airflow-lineage extract --dag-dir dags --var-dir variables
airflow-lineage query impact orders_ingestion_daily
airflow-lineage draw orders_ingestion_daily --open
```

---

## Why

Most lineage tooling needs a live scheduler, a plugin, or a running task to emit
events. That is useless when the question is asked *before* the change lands:

- Which DAGs break if I retire this one?
- What is the full upstream chain behind this table?
- Which tasks still point at that legacy JAR?
- Is my hand-drawn architecture diagram still true?

Static analysis answers all of those from source, in seconds, in CI, on a laptop,
on a branch that has never been deployed.

## What it does

| | |
|---|---|
| **Intra-DAG lineage** | `>>`, `<<`, `set_upstream/downstream`, `chain()`, list fan-out/fan-in |
| **Cross-DAG lineage** | `ExternalTaskSensor`, `TriggerDagRunOperator`, custom sensor classes, and producer/consumer edges inferred from `Variable.set`/`Variable.get` execution-time markers |
| **Dynamic tasks** | tasks built in `for` loops, comprehensions, factory functions, `range`/`enumerate`/`zip`, f-string and `.format()` task ids |
| **Config resolution** | reads a directory of exported Airflow Variable JSON files, so `Variable.get("cfg", deserialize_json=True)["tables"]` resolves to real values |
| **Conditional branches** | evaluates `if/elif/else` statically and captures only the branch that is actually active |
| **Task actions** | Databricks `main_class`, JAR, notebook path, python file, wheel package/entry point, parameters; dbt Cloud `job_id` |
| **Integrity audit** | fails loudly on phantom edges, disconnected dynamic tasks, self-loops, unresolved template ids |

### Truthfulness over completeness

A value that cannot be resolved statically is left `null` and surfaced under
`unresolved_references` — never guessed. Shape guards reject anything that looks
like leaked source text masquerading as a resolved value (`main_class` must be a
valid dotted class name; paths must not contain brackets or quotes). A wrong
"resolved" value is worse than an honest gap, especially when the output feeds a
migration inventory.

## Install

```bash
pip install airflow-lineage-toolkit                 # core: standard library only
pip install "airflow-lineage-toolkit[ui]"           # + local web UI (FastAPI)
pip install "airflow-lineage-toolkit[excel]"        # + .xlsx export (openpyxl)
```

Python 3.9+. The core extractor, query CLI, HTML visualizer, auditor and
CSV/JSON export have **zero runtime dependencies** — they run in any CI image.

## Quickstart

The repository ships a small synthetic Airflow project under `examples/` that
exercises every hard pattern, so you can see real output immediately:

```bash
airflow-lineage extract --dag-dir examples/dags --var-dir examples/variables
airflow-lineage audit
airflow-lineage query ecosystem
airflow-lineage draw --ecosystem --open
```

Point it at your own repository the same way:

```bash
airflow-lineage extract --dag-dir path/to/dags --var-dir path/to/variables
```

`--var-dir` is optional. Without it, values that come from Airflow Variables stay
unresolved (and are reported as such) — everything else still works.

### Exporting your Airflow Variables

The extractor reads a folder of JSON files shaped `{"<variable_name>": <value>}`,
which is what `airflow variables export` produces per key:

```bash
airflow variables export variables.json
```

Split it into one file per key, or drop the single file in the folder — both are
supported.

## Commands

```
airflow-lineage extract   Parse DAG files -> dag_lineage.json
airflow-lineage query     Ask dependency questions about the graph
airflow-lineage draw      Render an interactive HTML graph
airflow-lineage audit     Fail the build on graph-integrity errors
airflow-lineage export    Flatten to CSV / JSON / XLSX
airflow-lineage ui        Serve the local web UI
```

Useful queries:

```bash
airflow-lineage query impact <dag>            # blast radius, transitive
airflow-lineage query depends <dag>           # everything it waits for
airflow-lineage query path <from> <to>        # shortest route between two DAGs
airflow-lineage query hubs --top 10           # most-connected DAGs
airflow-lineage query cycles                  # circular dependencies
airflow-lineage query operator DatabricksSubmitRunOperator
airflow-lineage query find-task "load_.*"
airflow-lineage query export-mermaid <dag> --depth 2
```

Add `--json` to any query for machine-readable output. Full list: [docs/cli.md](docs/cli.md).

## Web UI

```bash
pip install "airflow-lineage-toolkit[ui]"
airflow-lineage ui --dag-dir examples/dags --var-dir examples/variables
```

`--dag-dir` and `--var-dir` default to `./dags` and `./variables` *relative to
wherever you run the command* — if that doesn't match your layout, the **Run
extraction** button in the UI will fail with `DAG directory not found`. Pass
the real paths explicitly, as above. Full options: [docs/cli.md](docs/cli.md#ui).

## Use it in CI

Catch a broken graph in review instead of at 3am:

```yaml
- run: pip install airflow-lineage-toolkit
- run: airflow-lineage extract --dag-dir dags --var-dir variables --quiet
- run: airflow-lineage audit          # non-zero exit on phantom/disconnected edges
```

## Use it as a library

```python
from airflow_lineage import build_lineage, LineageGraph

report = build_lineage("dags", "variables")
graph = LineageGraph(report)

print(graph.transitive_down("orders_ingestion_daily"))   # blast radius
print(graph.shortest_paths("raw_load", "exec_dashboard"))
```

The output document schema is described in [docs/lineage-schema.md](docs/lineage-schema.md).

## How it works

A per-file `ast.NodeVisitor` runs several passes: custom sensor classes, operator
factories, JSON payload builders, a fix-point over module-level assignments (so
forward references resolve), store-time Variable producers/consumers, and finally
the DAG/task/edge walk. Loops and comprehensions are *executed statically* against
resolved config so dynamically generated tasks land in the graph as concrete nodes
**with their edges**. See [docs/how-it-works.md](docs/how-it-works.md) for the
design notes and the failure modes it was hardened against.

## Limitations

- Task ids that depend on runtime state (XComs, `{{ ds }}`, API calls) cannot be
  resolved — they are reported as unresolved rather than invented.
- TaskFlow (`@task`) decorators are not yet mapped to task ids.
- Operator action detail currently covers Databricks and dbt Cloud; other
  operators are captured as nodes with their operator name only.

Contributions in these areas are very welcome — see
[CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT
