Metadata-Version: 2.4
Name: flowx-hpc
Version: 0.1.0a1
Summary: Python-native, restart-first workflow engine for scientific HPC computing
Author-email: Arsalan Akhtar <arsalan@atomira.eu>
License: GPL-3.0-only
License-File: LICENSE
Keywords: computational-chemistry,dft,hpc,materials-science,molecular-dynamics,scientific-computing,workflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Chemistry
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.11
Requires-Dist: ase>=3.22.0
Requires-Dist: click>=8.0
Requires-Dist: latex2mathml>=3.0
Requires-Dist: markdown>=3.4
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dateutil>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: snakemd>=2.0
Requires-Dist: spglib>=2.0
Requires-Dist: tabulate>=0.9
Requires-Dist: tomli-w>=1.0
Requires-Dist: typer>=0.13
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: cbor2>=5.0; extra == 'all'
Requires-Dist: kaleido>=0.2; extra == 'all'
Requires-Dist: msgpack>=1.0; extra == 'all'
Requires-Dist: mypy>=1.5; extra == 'all'
Requires-Dist: paramiko>=3.3; extra == 'all'
Requires-Dist: plotly>=5.0; extra == 'all'
Requires-Dist: pre-commit>=3.6; extra == 'all'
Requires-Dist: pymongo>=4.5; extra == 'all'
Requires-Dist: pytest-asyncio>=0.21; extra == 'all'
Requires-Dist: pytest-cov>=4.1; extra == 'all'
Requires-Dist: pytest-timeout>=2.1; extra == 'all'
Requires-Dist: pytest>=7.4; extra == 'all'
Requires-Dist: redis>=5.0; extra == 'all'
Requires-Dist: ruff>=0.16.4; extra == 'all'
Requires-Dist: sqlalchemy>=2.0; extra == 'all'
Requires-Dist: types-python-dateutil; extra == 'all'
Requires-Dist: types-pyyaml; extra == 'all'
Requires-Dist: weasyprint>=60; extra == 'all'
Provides-Extra: bandpath
Requires-Dist: pymatgen>=2024.1; extra == 'bandpath'
Requires-Dist: seekpath>=2.0; extra == 'bandpath'
Provides-Extra: dev
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: pre-commit>=3.6; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest-timeout>=2.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.16.4; extra == 'dev'
Requires-Dist: types-python-dateutil; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: fastio
Requires-Dist: cbor2>=5.0; extra == 'fastio'
Requires-Dist: msgpack>=1.0; extra == 'fastio'
Provides-Extra: images
Requires-Dist: kaleido>=0.2; extra == 'images'
Provides-Extra: plots
Requires-Dist: plotly>=5.0; extra == 'plots'
Provides-Extra: pymatgen
Requires-Dist: pymatgen>=2024.1; extra == 'pymatgen'
Provides-Extra: reports
Requires-Dist: weasyprint>=60; extra == 'reports'
Provides-Extra: seekpath
Requires-Dist: seekpath>=2.0; extra == 'seekpath'
Provides-Extra: sinks
Requires-Dist: pymongo>=4.5; extra == 'sinks'
Requires-Dist: sqlalchemy>=2.0; extra == 'sinks'
Provides-Extra: transports
Requires-Dist: boto3>=1.28; extra == 'transports'
Requires-Dist: paramiko>=3.3; extra == 'transports'
Requires-Dist: redis>=5.0; extra == 'transports'
Description-Content-Type: text/markdown

# FlowX

**Python-native, restart-first workflow engine for scientific HPC computing**

[![pipeline status](https://gitlab.com/siestastudio/flowx/badges/main/pipeline.svg)](https://gitlab.com/siestastudio/flowx/-/commits/main)
[![coverage report](https://gitlab.com/siestastudio/flowx/badges/main/coverage.svg)](https://gitlab.com/siestastudio/flowx/-/commits/main)
[![License](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Documentation](https://readthedocs.org/projects/flowx/badge/?version=latest)](https://flowx.readthedocs.io/en/latest/)

---

## Why FlowX?

FlowX is built for the reality of HPC scientific computing:
- **Air-gapped clusters** with strict schedulers
- **Preemption and walltime limits** that kill long jobs
- **Mixed environments** (Nix, modules, conda)
- **Filesystem as truth** (no database required)

### The FlowX Advantage

| Feature | FlowX | AiiDA | Jobflow |
|---------|-------|-------|---------|
| **Setup time** | 5 min | 90-180 min | 45 min |
| **Restart granularity** | Phase-level | Job-level | Job-level |
| **Database required** | No | Yes (PostgreSQL) | Optional |
| **Air-gap ready** | Yes | Limited | Limited |
| **Walltime handling** | Cooperative requeue | Manual | Manual |

---

## Quickstart

```bash
pip install git+https://gitlab.com/siestastudio/flowx.git
flowx configure siesta.command /path/to/siesta   # point FlowX at your engine
flowx pseudo download -e Si                      # fetch pseudopotentials
```

A single calculation:

```python
from flowx import run
from flowx.engines import siesta

job = siesta.scf("Si.cif", tier="standard")
result = run(job)

print(result.parsed.total_energy_eV)
print(result.parsed.scf_converged)
```

A workflow -- plain Python, one decorator:

```python
from flowx import run, task, workflow
from flowx.engines import siesta

@task
def relax_structure(structure, tier="standard"):
    job = siesta.relax(structure, tier=tier)
    return run(job)

@workflow
def study(structure):
    return relax_structure(structure)

result = study("Si.cif")
```

Then watch it from the shell:

```bash
flowx status                 # every run, with phase and state
flowx logs <run_id> -f       # follow output live
flowx workflow list          # workflows and their nodes
```

---

## Core Philosophy

### 1. Filesystem is Truth
Every run is a self-contained folder. No database required for correctness.

```
runs/relax-Si-20250130-143052/
├── FlowX-runspec.json       # Immutable job intent
├── FlowX-state.json         # Phase tracking (atomic, updated per phase)
├── FlowX-provenance.md      # Human-readable audit trail
├── FlowX-events.jsonl       # Append-only event log
├── FlowX-results.json       # Parsed results (typed schema)
├── siesta.fdf         # Generated engine input
├── siesta.out         # Raw program output
└── ...                # Other engine-native files (.XV, .DM, .bands, …)
```

### 2. Restart-First
Every phase checkpoints. Resume is trivial.

```bash
# Job gets killed after 3 hours? No problem.
flowx restart runs/relax-Si-20250130-143052
# Continues from last completed phase
```

**Time savings**: 5-50x on failures compared to restarting from scratch.

### 3. Decoupled Architecture
Submitter ≠ Runner ≠ Database. Air-gap friendly.

```
Laptop → JobBundle → Cluster Inbox → Runner → Results
```

### 4. Pure Python DSL
No YAML. No schema ceremony. Just Python.

```python
@task
def phonon_workflow(structure):
    relaxed = relax(structure)
    displacements = generate_displacements(relaxed["structure"])
    forces = [calculate_forces(d) for d in displacements["structures"]]
    return compute_phonons(relaxed["structure"], forces)
```

---

## Supported Engines

- **SIESTA** (native, primary engine)
- **Quantum ESPRESSO** (native)
- **VASP** (native)
- **exciting** (native) -- all-electron full-potential LAPW+lo, so its numbers
  share none of the pseudopotential approximations the other three rest on
- **GPAW** (transpiler skeleton -- no executor)
- **CP2K**, **CASTEP**, **ORCA** (planned)

All engines use a **Canonical Input Representation (CIR)** for portability:

```python
from flowx.cir import CanonicalInput, Structure, Kpoints, XC

cir = CanonicalInput(
    structure=Structure.from_file("Si.cif"),
    kpoints=Kpoints.gamma_centered(6, 6, 6),
    xc=XC.gga("PBE"),
    task="relax",
)

# Transpile to any engine
to_engine(cir, "siesta")  # → *.fdf files
to_engine(cir, "vasp")    # → INCAR, POSCAR, KPOINTS
to_engine(cir, "qe")      # → *.in files
# exciting takes a CIR without pseudopotentials; use exciting.scf() instead
```

---

## Phase Machine

Every job goes through deterministic phases with automatic checkpointing:

```
prepare → stage → run → harvest → validate → publish
```

- **prepare**: Generate engine inputs, resolve pseudopotentials
- **stage**: Ensure runtime environment, preflight checks
- **run**: Execute code with heartbeats and cooperative requeue
- **harvest**: Parse logs, collect outputs (idempotent)
- **validate**: Check success criteria
- **publish**: Push artifacts to external stores

**FlowX-state.json** is atomically updated after each phase. On crash, `flowx restart` re-enters the first incomplete phase.

---

## CLI

```bash
# Project
flowx init myproject                 # Initialize a new project
flowx configure --list               # Show resolved configuration

# Execution
flowx run <target>                   # Run a run_id, workflow_id, or .py script
flowx submit <wf_id> --cluster mn5   # Submit to an HPC scheduler
flowx restart <run_id>               # Restart from the last completed phase
flowx stop <run_id>                  # Graceful stop

# Monitoring
flowx status [<run_id>]              # Table of runs, or one run in detail
flowx logs <run_id> -f               # Follow output
flowx history <run_id>               # Execution timeline from the event log
flowx watch <wf_id>                  # Live workflow dashboard

# Analysis and housekeeping
flowx compare <run1> <run2>          # Diff two runs
flowx template list                  # The 20 workflow templates
flowx sweep                          # Classify the run store for retention
flowx archive <run_id>               # Archive to .tar.gz
```

`flowx --help` lists every command; most have short aliases (`ls`, `wf`, `cmp`).

---

## HPC & Air-Gap Operation

FlowX is designed for offline, pull-based execution:

1. **Prepare locally** → run directories with generated engine inputs
2. **Transfer** via rsync/SSH (`flowx submit <wf_id> --cluster <name> --zero-install`)
3. **The cluster-side coordinator executes** them (no inbound network required)
4. **Results written to the filesystem**, pulled back with `flowx download`

Nothing is required on the cluster but the engine itself: zero-install submit
ships a stdlib-only retry wrapper, and `--remote-python` moves the coordinator
onto the cluster so the campaign survives your laptop disconnecting.

---

## Installation

Not on PyPI yet. The distribution name is reserved as ``flowx-hpc`` -- the name
``flowx`` belongs to an unrelated package -- and the import name stays ``flowx``.

```bash
# From the repository
pip install git+https://gitlab.com/siestastudio/flowx.git

# With all optional dependencies
pip install "flowx-hpc[all] @ git+https://gitlab.com/siestastudio/flowx.git"

# Development installation
git clone https://gitlab.com/siestastudio/flowx.git
cd flowx
pip install -e '.[dev]'
```

---

## Documentation

**[flowx.readthedocs.io](https://flowx.readthedocs.io/en/latest/)**

- [Getting Started](https://flowx.readthedocs.io/en/latest/getting_started.html)
- [Workflows](https://flowx.readthedocs.io/en/latest/workflows.html)
- [CLI Reference](https://flowx.readthedocs.io/en/latest/cli.html)
- [Configuration](https://flowx.readthedocs.io/en/latest/configuration.html)
- [Engine APIs](https://flowx.readthedocs.io/en/latest/engines/) -- SIESTA, QE, VASP, exciting
- [Workflow Templates](https://flowx.readthedocs.io/en/latest/templates/)
- [API Reference](https://flowx.readthedocs.io/en/latest/api/)
- [Runnable examples](examples/) -- in this repository

Sources are under `docs/source/`; build them locally with `make -C docs html`.

---

## Development Status

**Current Phase**: v0.1.0 — Phases 1–3 complete

- Phase machine with phase-level restart + human-readable provenance
- Local runner, Slurm/PBS submitters, SSH transport, zero-install cluster submit
- Engines: SIESTA, Quantum ESPRESSO, VASP, exciting (GPAW transpiler skeleton)
- Workflow DAG: `@task`/`@workflow`, `parallel()`, `all_pairs()`, `portfolio()`, `loop_until()`, `flowx replay`
- 20 analysis templates (convergence, EOS, bands, phonons, NEB, defects, surfaces, …)
- Phase 4 (scale, AI, dashboard) and Phase 5 (CP2K, full GPAW, docs site) planned

---

## Contributing

We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

FlowX follows the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md).

---

## License

GNU General Public License v3.0 - see [LICENSE](LICENSE) for details.

### DFT engines are not included

FlowX drives external DFT codes; it does not contain, bundle or redistribute
any of them. Each engine must be obtained, installed and — where its own terms
require it — licensed by the user.

The engine adapters are interface code: they generate each code's native input
files and parse its output. No engine's source, binaries or pseudopotential
data are present in this repository. In particular, the VASP adapter assembles
a `POTCAR` by concatenating files from a pseudopotential library that the user
supplies and is licensed to hold; no VASP pseudopotentials are distributed here.

| Engine | Obtained from | Terms |
|---|---|---|
| SIESTA | [siesta-project.org](https://siesta-project.org) | GPL / open source |
| Quantum ESPRESSO | [quantum-espresso.org](https://www.quantum-espresso.org) | GPL / open source |
| exciting | [exciting-code.org](https://exciting-code.org) | GPL / open source |
| GPAW | [gpaw.readthedocs.io](https://gpaw.readthedocs.io) | GPL / open source |
| VASP | [vasp.at](https://www.vasp.at) | proprietary; user must hold a valid licence |

Pseudopotential libraries (PseudoDojo, SSSP, VASP PAW) are likewise downloaded
or supplied by the user under their own terms and are not redistributed.

---

## Citation

If you use FlowX in your research, please cite:

```bibtex
@software{flowx,
  title = {FlowX: Python-native workflow engine for scientific HPC},
  author = {Akhtar, Arsalan},
  year = {2026},
  version = {0.1.0},
  url = {https://gitlab.com/siestastudio/flowx}
}
```

---

## Support

- **Issues**: [GitLab Issues](https://gitlab.com/siestastudio/flowx/-/issues)
- **Chat**: [Discord](https://discord.gg/eJYCfj25Mt)
