Metadata-Version: 2.4
Name: py-data-engine
Version: 0.3.12
Summary: GUI orchestrator for python-based dataframe transform pipelines.
Author: Data Engine contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/bj-data-eng/data-engine
Project-URL: Repository, https://github.com/bj-data-eng/data-engine
Project-URL: Issues, https://github.com/bj-data-eng/data-engine/issues
Keywords: duckdb,excel,parquet,flow,polars,notebook
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Office/Business :: Financial :: Spreadsheet
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: APScheduler==3.11.2
Requires-Dist: duckdb==1.5.1
Requires-Dist: fastexcel==0.19.0
Requires-Dist: numpy==2.4.4
Requires-Dist: openpyxl==3.1.5
Requires-Dist: polars==1.39.3
Requires-Dist: pyarrow==23.0.1
Requires-Dist: PySide6==6.11.0
Requires-Dist: textual==8.2.3
Requires-Dist: xlsxwriter==3.2.9
Provides-Extra: polars-lts
Requires-Dist: polars-lts-cpu==1.33.1; extra == "polars-lts"
Provides-Extra: synthetic
Requires-Dist: openpyxl==3.1.5; extra == "synthetic"
Requires-Dist: pandas==3.0.2; extra == "synthetic"
Provides-Extra: notebook
Requires-Dist: ipykernel==7.2.0; extra == "notebook"
Requires-Dist: ipython==9.12.0; extra == "notebook"
Requires-Dist: jupyterlab==4.5.7; extra == "notebook"
Requires-Dist: notebook==7.5.6; extra == "notebook"
Provides-Extra: docs
Requires-Dist: myst-parser==5.0.0; extra == "docs"
Requires-Dist: Sphinx==9.1.0; extra == "docs"
Requires-Dist: sphinx_rtd_theme==3.1.0; extra == "docs"
Provides-Extra: package
Requires-Dist: build==1.4.2; extra == "package"
Requires-Dist: pyinstaller==6.19.0; extra == "package"
Requires-Dist: twine==6.2.0; extra == "package"
Provides-Extra: test
Requires-Dist: openpyxl==3.1.5; extra == "test"
Requires-Dist: pandas==3.0.2; extra == "test"
Requires-Dist: pydocstyle==6.3.0; extra == "test"
Requires-Dist: pytest==9.0.3; extra == "test"
Requires-Dist: pytest-cov==7.1.0; extra == "test"
Provides-Extra: dev
Requires-Dist: ipykernel==7.2.0; extra == "dev"
Requires-Dist: ipython==9.12.0; extra == "dev"
Requires-Dist: jupyterlab==4.5.7; extra == "dev"
Requires-Dist: myst-parser==5.0.0; extra == "dev"
Requires-Dist: notebook==7.5.6; extra == "dev"
Requires-Dist: numpy==2.4.4; extra == "dev"
Requires-Dist: openpyxl==3.1.5; extra == "dev"
Requires-Dist: pandas==3.0.2; extra == "dev"
Requires-Dist: pydoclint==0.8.3; extra == "dev"
Requires-Dist: pydocstyle==6.3.0; extra == "dev"
Requires-Dist: pyinstaller==6.19.0; extra == "dev"
Requires-Dist: pytest==9.0.3; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: ruff==0.15.10; extra == "dev"
Requires-Dist: Sphinx==9.1.0; extra == "dev"
Requires-Dist: sphinx_rtd_theme==3.1.0; extra == "dev"
Requires-Dist: viztracer==1.1.1; extra == "dev"
Dynamic: license-file

# Data Engine

Data Engine is a GUI orchestrator for Python dataframe workflows. It lets you
author flows as plain Python modules, run them manually or automatically, and
inspect parquet-first outputs from the desktop app.

The runtime is built around:

- workspace-based flow discovery
- manual, poll, and schedule execution modes
- Polars and DuckDB-friendly flow steps
- mirrored output paths for source-driven runs
- saved run, log, and dataframe inspection state
- desktop and terminal operator surfaces

## Install

Use the installer for your environment:

- macOS: [INSTALL/INSTALL MAC.command](INSTALL/INSTALL%20MAC.command)
- Windows: [INSTALL/INSTALL WINDOWS.bat](INSTALL/INSTALL%20WINDOWS.bat)
- Windows VM / CPU-safe Polars path: [INSTALL/INSTALL WINDOWS_VM.bat](INSTALL/INSTALL%20WINDOWS_VM.bat)

For local development:

```bash
python -m pip install --constraint requirements/constraints.txt -e ".[dev]"
```

For a published package install:

```bash
python -m pip install py-data-engine
```

Data Engine requires Python `>=3.14`.

Dependency pinning, constrained installs, and hash-locked runtime installs are
documented in [SECURITY.md](SECURITY.md).

## Start

Desktop GUI:

```bash
data-engine start gui
```

Terminal UI:

```bash
data-engine start tui
```

Headless commands:

```bash
data-engine list
data-engine show example_summary
data-engine run --once example_summary
data-engine run
```

## Minimal Flow

```python
from data_engine import Flow
import polars as pl


def read_docs(context):
    return pl.read_excel(context.source.path)


def keep_open(context):
    return context.current.filter(pl.col("status") == "OPEN")


def write_parquet(context):
    output = context.mirror.with_suffix(".parquet")
    context.current.write_parquet(output)
    return output


def build():
    return (
        Flow(group="Docs")
        .watch(
            mode="poll",
            source="../../../example_data/Input/docs_flat",
            interval="5s",
            extensions=[".xlsx", ".xls", ".xlsm"],
            settle=1,
        )
        .mirror(root="../../../example_data/Output/example_mirror")
        .step(read_docs, save_as="raw_df")
        .step(keep_open, use="raw_df", save_as="filtered_df")
        .step(write_parquet, use="filtered_df")
    )
```

Each authored flow module exports `build() -> Flow`. The module filename is the
flow identity.

## Workspaces

Data Engine discovers workspaces from a collection root. Each workspace keeps
authored flows under:

```text
workspaces/<workspace_id>/flow_modules/
```

Shared workspace state lives under:

```text
workspaces/<workspace_id>/.workspace_state/
```

Machine-local runtime artifacts are stored outside the authored workspace.

## Useful APIs

```python
from data_engine import Flow, FlowContext, discover_flows, load_flow, run
```

Common `Flow` methods:

- `.watch(...)`
- `.mirror(...)`
- `.date_range_input(...)`
- `.step(...)`
- `.collect(...)`
- `.map(...)`
- `.step_each(...)`
- `.preview(...)`
- `.run_once()`
- `.run()`

Common `FlowContext` values:

- `context.source`
- `context.mirror`
- `context.current`
- `context.objects`
- `context.metadata`
- `context.database("analytics.duckdb")`
- `context.template("reports/base.xlsx")`
- `context.debug`

The full authoring guide and helper reference live in
`src/data_engine/docs/sphinx_source/guides/`.

## Testing

```bash
python -m pytest -q
python -m build
python -m twine check dist/*
```

## Status

This project is pre-alpha. Internal architecture is still moving quickly, and
backwards compatibility is not a current goal.
