Metadata-Version: 2.4
Name: pantarei
Version: 0.11.1
Summary: A general-purpose workflow manager
Author-email: Daniele Coslovich <daniele.coslovich@umontpellier.fr>
License: GPLv3
Project-URL: repository, https://framagit.org/coslo/pantarei
Project-URL: homepage, https://framagit.org/coslo/pantarei
Project-URL: documentation, https://coslo.frama.io/pantarei
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Development Status :: 4 - Beta
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: tinydb
Requires-Dist: dill
Requires-Dist: pyyaml
Dynamic: license-file

# Pantarei

[![pypi](https://img.shields.io/pypi/v/pantarei.svg)](https://pypi.python.org/pypi/pantarei/)
[![version](https://img.shields.io/pypi/pyversions/pantarei.svg)](https://pypi.python.org/pypi/pantarei/)
[![license](https://img.shields.io/pypi/l/pantarei.svg)](https://en.wikipedia.org/wiki/GNU_General_Public_License)
[![pipeline status](https://framagit.org/coslo/pantarei/badges/master/pipeline.svg)](https://framagit.org/coslo/pantarei/-/commits/master)
[![coverage report](https://framagit.org/coslo/pantarei/badges/master/coverage.svg)](https://framagit.org/coslo/pantarei/-/commits/master)

A general-purpose workflow manager - because *everything* flows

```bash
$ rei run project.py 
$ rei ls -l project.py 
⬜ project.py  |███████████         | 50% [2/4] 0:00:14
✅ ended    50% [2/4]
⭐ running  50% [2/4]
   ✅ f/185e9a70bf662355983d79881877056a 0:00:05 f(x=0)
   ✅ f/216ec88e5485e1ae439c37d3f94cab8f 0:00:05 f(x=1)
   ⭐ f/4e17f7f9fc6073c3d511bc12c7b6722a 0:00:02 f(x=2)
   ⭐ f/1853daf3a769344e10cfcbdef036a8c9 0:00:02 f(x=3)
```

## Quick start

Pantarei builds on four kinds of execution units:

- **functions**: stateless, Python callables
- **tasks**: wrapped functions that cache execution results
- **threads**: wrapped tasks for batch execution in /shared-memory/ parallel environments
- **jobs**: wrapped tasks for batch execution in /distributed-memory/ parallel environments

As a catch-all, it also provides a **parallel** execution unit that uses whatever scheduling system is available on your environment, defaulting to shared-memory parallelism.

To see it in action, say you have a Python function
```python
def f(x):
    import time
    time.sleep(2)
    return x
```

Wrap the function with a Task and call it with a range of arguments
```python
from pantarei import Task

task = Task(f)
for x in [1, 2]:
    task(x=x)
```

The task's results are cached: a successive execution will just fetch the results (like `joblib`)
```python
results = task(x=1)
```

We wrap the task with `Parallel` and submit its execution to a local scheduler, such as  `SLURM`, or to a multi-threading scheduler as fallback
```python
from pantarei import Parallel

job = Parallel(task)
for x in [3, 4]:
    job(x=x)
```

If you want to get the jobs' results, wait until they are done
```python
job.scheduler.wait()
results = job(x=3)
```

## Command line interface

Pantarei comes with a command line interface to run and manage jobs. If you like working from the terminal, you'll find yourself at home.

Run a script with some jobs
```bash
rei run script.py
```

Check the status of the jobs
```bash
rei ls -l script.py
```
```
🟩 script.py |████████████████████| 0:00:00 [2/2]
   ✅ f/3de5a949fa3c880e35165fc6820ce82e 0:00:00 f(x=1)
   ✅ f/26766c2fda253b7aeb1adaa02f31e93b 0:00:00 f(x=2)
```

Inspect the jobs' cache (metadata and results)
```bash
rei cat script.py
```

Clear the jobs' cache and artifacts
```bash
rei rm -rf script.py
```

There is much more of course: `rei --help` and `rei <command> --help` are your best friends.

## Documentation

Check out the [tutorial](https://coslo.frama.io/pantarei) for more examples and the [public API](https://coslo.frama.io/pantarei) for full details.

The CLI interface (`rei`) is documented via its own help pages.

## Installation

From pypi
```
pip install pantarei
```

## TODO

- [ ] submit on remote cluster
- [ ] handle task dependencies
- [ ] add Workflow / Queue
- [ ] perhaps add signac-like view() or checkout() method to check out a view of cache as folders

## Contributing

Contributions to the project are welcome. If you wish to contribute, check out [these guidelines]().

## Authors

- Daniele Coslovich
