Metadata-Version: 2.4
Name: exa-atow-compiler
Version: 0.1.1
Summary: Compiler for generating executable CWL workflows and Galaxy tools from HAPS descriptions
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cwltool
Requires-Dist: PyYAML
Requires-Dist: cwl2nx
Requires-Dist: graph2cwl
Requires-Dist: ebsclient
Requires-Dist: ebstemplate
Requires-Dist: exa-atow-runtime<0.2.0,>=0.1.0
Requires-Dist: mcp<3,>=2
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# Exa-AToW Compiler

The Exa-AToW Compiler generates executable workflow components from HAPS descriptions for three target environments:

- **Galaxy**: Galaxy tool XML files and Python proxies
- **CWL**: an executable CWL workflow, CWL task descriptions, and Python proxies
- **MCP**: MCP servers that expose HAPS applications as MCP tools

Generated components use the separate `exa-atow-runtime` package to communicate with eBservice/eBuffer, submit HAPS jobs, monitor execution, recover supported jobs, and process results.

## Contents

- [How it works](#how-it-works)
- [Requirements](#requirements)
- [Installation](#installation)
- [Quick start](#quick-start)
- [HAPS descriptions](#haps-descriptions)
- [CWL mode](#cwl-mode)
- [Galaxy mode](#galaxy-mode)
- [MCP mode](#mcp-mode)
- [Runtime and recovery](#runtime-and-recovery)
- [Examples and Makefile](#examples-and-makefile)
- [Tests](#tests)
- [Project structure](#project-structure)
- [Development and release](#development-and-release)
- [Security](#security)
- [License](#license)

## How it works

The compiler and runtime have separate responsibilities:

| Component | Responsibility | Typical installation location |
|---|---|---|
| `exa-atow-compiler` | Reads HAPS descriptions and generates Galaxy, CWL, or MCP artifacts | Developer workstation or workflow-generation environment |
| `exa-atow-runtime` | Executes the shared HAPS job-submission, monitoring, recovery, and result-handling logic | Environment in which a generated proxy or MCP server runs |

The general flow is:

1. The compiler reads one or more HAPS XML descriptions.
2. In CWL mode, it also reads an abstract workflow and a CWL job-input file.
3. It generates artifacts for the selected target environment.
4. A workflow engine, Galaxy, or an MCP client starts the generated component.
5. The generated component calls `exa-atow-runtime` to interact with eBservice/eBuffer and the remote HAPS job.

## Requirements

The compiler requires:

- Python 3.12 or newer
- Access to the HAPS descriptions to compile

Executing generated components additionally requires:

- Access to a configured eBservice/HAPS deployment
- Valid HAPS/eBservice credentials
- Access to the selected HPC centre

The runtime is distributed separately as `exa-atow-runtime` and supports Python 3.11 or newer.

## Installation

### Install from PyPI

Create and activate a virtual environment:

```bash
python3.12 -m venv .venv
source .venv/bin/activate
```

Install the compiler:

```bash
python -m pip install exa-atow-compiler
```

This installs the compiler, a compatible `exa-atow-runtime`, the MCP SDK, and the remaining compiler dependencies.

Confirm the installation:

```bash
exa-atow-compiler --help
```

The command exposes three modes:

```text
exa-atow-compiler galaxy ...
exa-atow-compiler cwl ...
exa-atow-compiler mcp ...
```

### Install only the runtime

An execution environment that does not generate artifacts can install only the runtime:

```bash
python -m pip install exa-atow-runtime
```

This is useful for Galaxy tool environments and other systems that execute already-generated proxies.

### Install from a source checkout

From the repository root:

```bash
python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install -e runtime
python -m pip install -e ".[test]"
python -m pip install planemo
```

The local runtime is installed first because the compiler declares it as a dependency.

The project Makefile provides the same development setup:

```bash
make install
```

Useful discovery commands are:

```bash
make help
make list-examples
make show-config
```

## Quick start

### Generate Galaxy tools

```bash
exa-atow-compiler galaxy \
    --haps tests/galaxy-tools/echo-uppercase/echo.xml \
           tests/galaxy-tools/echo-uppercase/uppercase.xml \
    --output_dir tests/galaxy-tools/echo-uppercase/generated
```

### Generate an executable CWL workflow

```bash
exa-atow-compiler cwl \
    --workflow tests/CWL/echo-uppercase/workflow.cwl \
    --haps tests/CWL/echo-uppercase/echo.xml \
           tests/CWL/echo-uppercase/uppercase.xml \
    --job tests/CWL/echo-uppercase/job.yml \
    --data-logistics-haps tests/CWL/echo-uppercase/data_logistics.xml \
    --output_dir tests/CWL/echo-uppercase/generated
```

`--data-logistics-haps` is needed when a dependency connects tasks assigned to different HPC centres.

### Generate MCP servers

```bash
exa-atow-compiler mcp \
    --haps tests/MCP/echo.xml tests/MCP/uppercase.xml \
    --output_dir mcp_generated
```

## HAPS descriptions

A HAPS XML description is an Extended Galaxy Tool Description (XGTD). It combines the user-facing tool interface with the information required to invoke a remote HAPS application.

Depending on the application, the `<application>` element can define:

- Application name and MIME type
- Application arguments and results
- eBin input names
- eBout output names
- Runtime or policy identifiers
- Application tags

The description can also define tool inputs, outputs, output mappings, help text, and the HPC centres supported by the application.

At execution time:

- Local inputs mapped to eBins are uploaded through eBuffer.
- Ordinary application parameters are passed as remote job arguments.
- Application results can be mapped to local JSON metadata.
- eBout outputs can be downloaded and exposed as local workflow outputs.

## CWL mode

### Inputs

CWL mode accepts:

- An abstract CWL workflow
- One HAPS description for every remote workflow task
- A CWL job-input file
- An optional data-logistics HAPS description
- An output directory

Show the complete CLI reference with:

```bash
exa-atow-compiler cwl --help
```

### CWL job-input file

The job file provides workflow inputs, HAPS credentials, and the HPC centre selected for each remote task.

Example:

```yaml
message:
  class: File
  path: message
username: your_ebservice_username
password: your_ebservice_password
echo_hpc_center: idris
uppercase_hpc_center: idris
```

For a step named `echo`, the corresponding centre-selection input is `echo_hpc_center`. For a step named `uppercase`, it is `uppercase_hpc_center`.

Never commit real credentials in a job file.

### Generated artifacts

For the Echo-to-Uppercase example, CWL mode generates:

```text
generated/
├── wf.yaml
├── echo.cwl
├── echo_proxy.py
├── uppercase.cwl
└── uppercase_proxy.py
```

When a cross-centre dependency is detected, the compiler additionally generates the required data-logistics task and proxy.

The generated `wf.yaml` is the executable CWL workflow. Each remote step is represented by a generated `CommandLineTool` and a Python proxy.

### Intermediate data dependencies

Dependent HAPS tasks exchange local JSON metadata rather than routing remote intermediate data through the local CWL engine.

An intermediate metadata document can contain:

```json
{
  "remote_path": "/remote/path/to/output.txt",
  "status": "completed",
  "hpc_center": "idris"
}
```

The downstream proxy reads the metadata and passes the remote path to the next HAPS job. This allows intermediate data to remain on the HPC infrastructure.

### Cross-centre data logistics

Tasks assigned to the same centre are connected directly:

```text
echo -> uppercase
```

If dependent tasks are assigned to different centres, the compiler inserts a data-logistics step:

```text
echo -> data_logistics_echo_to_uppercase -> uppercase
```

The inserted step transfers the remote data and produces updated metadata for the consuming task.

### Validate the generated workflow

```bash
python -m cwltool \
    --validate tests/CWL/echo-uppercase/generated/wf.yaml
```

### Execute a generated workflow

Generation and execution are separate operations. Execute a logical workflow instance from its own run directory:

```bash
cd tests/CWL/echo-uppercase/run1
make run
```

A run directory contains a Makefile, an `.exa_atow_instance` file, and a results directory:

```text
run1/
├── Makefile
├── .exa_atow_instance
└── results/
```

The first `make run` creates the instance identifier when it does not already exist. Repeating `make run` in the same directory reuses the identifier and resumes the same logical workflow instance.

Create another run directory for an independent execution. Do not share an instance identifier between independent workflow runs.

### CWL recovery

CWL recovery allows a restarted `cwltool` process to reconnect to remote HAPS jobs that were submitted by the same logical workflow instance.

The run Makefile exports and preserves:

```text
EXA_ATOW_INSTANCE_ID
EXA_ATOW_STATE_DIR
```

The default state root is:

```text
~/.exa_atow/
```

Task state is stored under:

```text
$EXA_ATOW_STATE_DIR/instances/<instance-id>/<task-id>.json
```

The state can contain the workflow instance ID, task ID, HPC centre, microservice UUID, submission attempt, remote job UUID, remote status, and recovery tag.

When a proxy restarts, it first attempts recovery using the saved remote job UUID. If interruption occurred after remote submission but before the UUID was saved locally, it can search using the recovery tag associated with that submission attempt.

Typical decisions are:

| Saved or recovered state | Action |
|---|---|
| No reusable state | Submit a new job |
| Initialized, submitting, pending, or running | Recover and continue monitoring |
| Completed | Recover and recreate the required local outputs |
| Failed, cancelled, stopped, or expired | Submit a new attempt |

A completed job is recovered so that a restarted `cwltool` process can recreate its local CWL outputs without repeating the remote computation.

If the saved job cannot be found, or if its remote outputs are no longer available, the task is marked as expired and a new attempt is submitted. Each new attempt receives a new recovery tag.

## Galaxy mode

### Inputs and generation

Galaxy mode accepts one or more HAPS descriptions and an output directory:

```bash
exa-atow-compiler galaxy \
    --haps tests/galaxy-tools/echo-uppercase/echo.xml \
           tests/galaxy-tools/echo-uppercase/uppercase.xml \
    --output_dir tests/galaxy-tools/echo-uppercase/generated
```

The generated directory contains one Galaxy XML description and one Python proxy for each HAPS application:

```text
generated/
├── echo.xml
├── echo_proxy.py
├── uppercase.xml
└── uppercase_proxy.py
```

The XML file defines the Galaxy-facing tool interface and invokes the corresponding generated proxy.

You can generate a configured example through the Makefile:

```bash
make compile-galaxy GALAXY_EXAMPLE=echo-uppercase
```

### Runtime requirement in Galaxy

Galaxy must be able to import `exa_atow_runtime` in the Python environment used to run the generated proxies.

For a packaged installation:

```bash
/path/to/galaxy/.venv/bin/python -m pip install exa-atow-runtime
```

For local development:

```bash
/path/to/galaxy/.venv/bin/python -m pip install -e \
    /path/to/exa-atow-compiler/runtime
```

Verify the installation:

```bash
/path/to/galaxy/.venv/bin/python -c \
    "import exa_atow_runtime; print(exa_atow_runtime.__file__)"
```

Galaxy does not need the complete compiler package when it only executes generated tools.

### Install generated tools in Galaxy

Generating a Galaxy tool does not install it in a Galaxy server. After generation:

1. Copy or link the generated files into the Galaxy tools directory.
2. Keep every XML file beside its corresponding proxy.
3. Register the XML files in `tool_conf.xml` or another Galaxy tool configuration file.
4. Restart Galaxy or reload its toolbox configuration.
5. Confirm that the tools appear in the Galaxy tool panel.

Example layout:

```text
<GALAXY_ROOT>/tools/exa_atow/echo-uppercase/
├── echo.xml
├── echo_proxy.py
├── uppercase.xml
└── uppercase_proxy.py
```

### Planemo profile and Galaxy API key

Planemo uses a Galaxy API key to communicate with the Galaxy server.

Set the API key in the current shell:

```bash
export GALAXY_API_KEY="your-api-key"
```

Create and inspect the configured Planemo profile:

```bash
make galaxy-profile-create
make galaxy-profile-list
```

The default development configuration uses:

```text
Profile: local-galaxy
URL:     http://localhost:8080
```

### Create and execute a Galaxy workflow

Build a workflow through the Galaxy interface using the generated tools, mark the required final outputs, and export it to the corresponding example directory as `workflow.ga`.

Create a workflow job-input file:

```bash
make galaxy-job-init GALAXY_EXAMPLE=echo-uppercase
```

Edit the generated `workflow-job.yml` before running. Input paths are relative to the directory containing this file.

Lint and execute the exported workflow:

```bash
make galaxy-workflow-lint GALAXY_EXAMPLE=echo-uppercase
make run-galaxy-workflow GALAXY_EXAMPLE=echo-uppercase
```

The returned Galaxy invocation ID belongs to Galaxy.

Rerun failed Galaxy jobs belonging to an existing invocation:

```bash
make rerun-galaxy-workflow \
    GALAXY_EXAMPLE=echo-uppercase \
    INVOCATION_ID=<invocation-id>
```

Download the workflow outputs:

```bash
make galaxy-output-download \
    GALAXY_EXAMPLE=echo-uppercase \
    INVOCATION_ID=<invocation-id>
```

Downloaded results are stored under:

```text
tests/galaxy-tools/echo-uppercase/run-output/<invocation-id>/
```

The directory also contains `manifest.json`, describing the downloaded outputs.

## MCP mode

MCP mode generates servers that expose HAPS applications as tools to MCP-capable AI clients.

### Generate servers

```bash
exa-atow-compiler mcp \
    --haps tests/MCP/echo.xml tests/MCP/uppercase.xml \
    --output_dir mcp_generated
```

Using the Makefile:

```bash
make compile-mcp \
    MCP_HAPS="tests/MCP/echo.xml tests/MCP/uppercase.xml"
```

For a HAPS application named `echo`, the generated file is:

```text
mcp_generated/echo_mcp_server.py
```

### Generated tool interface

The generated tool parameters are derived from the HAPS inputs. The HPC-centre parameter is restricted to the centres declared in the description:

```python
def echo(
    message: str,
    hpc_center: Literal["idris", "cines", "tgcc"],
) -> dict:
    ...
```

HAPS credentials are runtime configuration and are not exposed as MCP tool arguments.

Set them in the MCP server environment:

```bash
export EXA_ATOW_USERNAME="your-username"
export EXA_ATOW_PASSWORD="your-password"
```

### Run a generated server

If the server runs in the same environment as the compiler installation, its runtime and MCP dependencies are already installed:

```bash
python mcp_generated/echo_mcp_server.py
```

If a generated server is copied into a separate environment, install its execution dependencies there:

```bash
python -m pip install exa-atow-runtime "mcp>=2,<3"
```

Then configure the MCP client to start the generated Python file using the interpreter from that environment.

## Runtime

The runtime package is located under `runtime/` in the source tree and is published independently as `exa-atow-runtime`.

Generated components import shared functions from:

```python
from exa_atow_runtime import ...
```

The runtime centralizes:

- eBservice/eBuffer client creation
- Remote microservice discovery or creation
- HAPS job submission
- Job-status monitoring
- Persistent state and supported recovery paths
- HAPS result processing
- JSON metadata generation
- eBout downloading


## Examples and Makefile

CWL examples are discovered under:

```text
tests/CWL/<example>/
```

Galaxy examples are discovered under:

```text
tests/galaxy-tools/<example>/
```

Each example has an `example.mk` file that identifies its HAPS descriptions and other example-specific configuration.

Common targets include:

```bash
make list-examples
make show-config

make compile-cwl CWL_EXAMPLE=echo-uppercase
make compile-galaxy GALAXY_EXAMPLE=echo-uppercase
make compile-mcp MCP_HAPS="tests/MCP/echo.xml"
```

The compiler targets generate artifacts; they do not implicitly execute a CWL workflow or install a generated Galaxy tool.

## Tests

Run the complete test suite:

```bash
python -m pytest
```

or:

```bash
make test
```

The test suite covers CWL, Galaxy, and MCP generation:

```bash
python -m pytest tests/test_cwl_generation.py
python -m pytest tests/test_galaxy_generation.py
python -m pytest tests/test_mcp_generation.py
```

## Cleaning

Remove generated artifacts and Python caches:

```bash
make clean
```

Remove generated artifacts and the development virtual environment:

```bash
make clean-all
```

Persistent state under `~/.exa_atow/` is outside the repository and is not removed by these targets.

## Project structure

```text
exa-atow-compiler/
├── src/
│   └── exa_atow_compiler/
│       ├── __init__.py
│       ├── compiler.py
│       ├── galaxy_tool_generator.py
│       ├── mcp_generator.py
│       ├── proxy_generator.py
│       ├── task_cwl_generator.py
│       └── templates/
│           └── proxy_task.py.tmpl
│
├── runtime/
│   ├── exa_atow_runtime/
│   │   ├── __init__.py
│   │   └── runtime.py
│   ├── LICENSE
│   ├── pyproject.toml
│   └── README.md
│
├── tests/
│   ├── CWL/
│   ├── MCP/
│   ├── galaxy-tools/
│   ├── test_cwl_generation.py
│   ├── test_galaxy_generation.py
│   └── test_mcp_generation.py
│
├── .gitignore
├── LICENSE
├── Makefile
├── pyproject.toml
└── README.md
```

Generated directories, run outputs, caches, virtual environments, and build distributions are ignored by Git.

## Development and release
The following instructions are intended for Exa-AToW developers and package maintainers. Users installing the compiler from PyPI do not need these steps.
Install the two local projects in editable mode:

```bash
python -m pip install -e runtime
python -m pip install -e ".[test]"
```

Build and validate the runtime first:

```bash
python -m build --outdir runtime/dist-release runtime
python -m twine check runtime/dist-release/*
```

Then build and validate the compiler:

```bash
python -m build --outdir dist-release .
python -m twine check dist-release/*
```

Publish releases in this order:

1. `exa-atow-runtime`
2. `exa-atow-compiler`

The compiler depends on a compatible runtime release, so the runtime must be available to the package index first.

## Security

Do not commit or publish:

- HAPS/eBservice usernames or passwords
- Galaxy API keys
- PyPI API tokens
- `.env` files
- Private infrastructure credentials

Use environment variables or the appropriate secret-management mechanism for the execution environment.

Before building a public release, inspect the source distribution and wheel to ensure they contain only the intended source, templates, README, and licence files.

## License

Exa-AToW Compiler and Exa-AToW Runtime are distributed under the MIT License.

Copyright (c) 2026 Sara Sadat Hoseininasab, Université de Rennes, Exa-AToW.

See `LICENSE` for the full licence text.
