Metadata-Version: 2.4
Name: mlpipelineholder
Version: 0.2.2
Summary: Lightweight machine learning pipeline handler
License-Expression: MIT
License-File: LICENSE
Keywords: machine-learning,pipeline,experimentation,orchestration
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Provides-Extra: all
Provides-Extra: dataframe
Provides-Extra: memory
Provides-Extra: torch
Requires-Dist: dask[dataframe] ; extra == "all"
Requires-Dist: dask[dataframe] ; extra == "dataframe"
Requires-Dist: numpy
Requires-Dist: pandas ; extra == "all"
Requires-Dist: pandas ; extra == "dataframe"
Requires-Dist: psutil ; extra == "all"
Requires-Dist: psutil ; extra == "memory"
Requires-Dist: pyarrow ; extra == "all"
Requires-Dist: pyarrow ; extra == "dataframe"
Requires-Dist: termcolor
Requires-Dist: torch ; extra == "all"
Requires-Dist: torch ; extra == "torch"
Project-URL: Homepage, https://github.com/XiaokaiCui/MLPipelineHolder
Project-URL: Issues, https://github.com/XiaokaiCui/MLPipelineHolder/issues
Project-URL: Repository, https://github.com/XiaokaiCui/MLPipelineHolder
Description-Content-Type: text/markdown

# MLPipelineHolder

MLPipelineHolder is a lightweight Python library for building, running, tracking, and modifying experiment-driven machine-learning pipelines, particularly in Jupyter and Google Colab notebooks.

## Installation

Install from PyPI:

```bash
pip install mlpipelineholder
```

Install optional integrations as needed:

```bash
pip install "mlpipelineholder[dataframe]"
pip install "mlpipelineholder[torch]"
pip install "mlpipelineholder[memory]"
pip install "mlpipelineholder[all]"
```

Each extra adds:

- `dataframe`: pandas, PyArrow, and Dask DataFrame support
- `torch`: PyTorch model, tensor, and optimiser persistence
- `memory`: `psutil`-based memory profiling logs
- `all`: all optional features listed above

The core package requires Python 3.11 or later and includes `termcolor` and NumPy.

## At a glance

MLPipelineHolder organises workflows into explicit execution blocks and nested child pipelines while keeping their runtime structure easy to inspect and modify. It helps you:

- Organise data flows and manage variables, configurations, outputs, and dependencies through clearly defined scopes.
- Persist intermediate and final outputs to disk, then reload them quickly and easily after a kernel restart.
- Automatically run independent functions in parallel within the same execution block, while keeping cross-block execution explicit and ordered by priority.
- Reduce RAM usage by storing large artifacts on disk without sacrificing pipeline usability. Enable `memory_saving_mode` to release objects that are no longer needed.
- Track logs, results, and pipeline state with minimal effort.
- Improve the stability and reproducibility of modelling and analysis outputs while retaining full flexibility over the pipeline structure.

## Fastest way to start: collaborate with an LLM on your notebook

If you already have a data modelling or analysis notebook, the fastest way to get started is to ask an LLM to convert it into a pipeline-managed workflow.

This repository includes a notebook-oriented guide for the LLM agent:

- [`PIPELINE_CREATOR_SKILL.md`](PIPELINE_CREATOR_SKILL.md)

The guide explains how to inspect your notebook, design pipeline scopes, identify persistence and memory requirements, and produce the converted code.

### Low-pressure onboarding workflow

You do not need a perfect prompt or complete answers. Responses such as "not sure", "suggest for me", or "use sensible defaults" are fine.

The agent will inspect your code, propose a scope plan and likely disk-backed outputs, and ask only a short batch of blocking or high-impact questions.

### Recommended prompt template

Copy and paste this prompt, filling in what you can:

```text
I have a Jupyter notebook for data modelling or analysis that I want to convert into a pipeline-managed notebook using MLPipelineHolder.

Repository:
https://github.com/XiaokaiCui/MLPipelineHolder

Please read the repository and the file `PIPELINE_CREATOR_SKILL.md` first. Then use that guidance to convert my notebook.

Here are my workflow details (rough answers are fine, or say "not sure" / "suggest for me"):
- Group 1 (Intent and Context): Are we creating a new pipeline from scratch, or extending an existing saved parent pipeline? If extending, what is the parent path or structure?
- Group 2 (Scope and Prefix): Which parts are shared vs workflow-local, and what prefix should we use for workflow-local config/values?
- Group 3 (Large Objects): Which produced objects are expected to be large and should use save_to_disk? (Answer yes/no/not sure; suggest candidates for me.)
- Group 4 (Memory Settings): Do you want memory_saving_mode or memory_profile_logging enabled? (Answer yes/no/not sure; suggest defaults for me.)
- Group 5 (Persistence and Callables): Do you intend to save and reload the pipeline later, and where are your callables defined?

Please inspect my code, propose likely large outputs and a scope plan, and then ask only a short batch of blocking or high-impact questions before writing the code.

Your final output must include:
1. Structure Explanation: A short explanation of the proposed pipeline structure.
2. Concise Conversion Manifest: The manifest detailing hierarchy, stage types, priorities, inputs/outputs, prefixes, and disk-backed outputs.
3. Converted Code: The complete converted notebook code as one contiguous deliverable. Use one fenced code block if the script is reasonably short; otherwise, create one `.py` file and provide its path or link. Do not split the script into blocks that I must combine manually.
4. Assumptions and Questions: Any assumptions made or blocking questions.
5. Verification and Persistence Notes: Notes on how to verify the pipeline and any persistence limitations.

I will now provide my notebook/code.
```

### Understanding large objects and memory settings

**Large objects and disk backing**

Large objects are usually large DataFrames, arrays, models, or expensive intermediates. Storing them on disk lowers the RAM retained during execution, but it adds serialisation and I/O cost. The `save_to_disk` option applies only to declared produced outputs of a block, not to arbitrary inputs set via `set_value`.

**Memory saving and profiling**

- `memory_saving_mode` performs best-effort cleanup of intermediate values after each block finishes.
- `memory_profile_logging` logs memory usage after computation and cleanup for each block.

Attached child pipelines inherit both settings from their parent, so configure the policy on the owning or root pipeline.

## Core concepts

### 1. PipelineHandler

Construct with:

- `registration_name`
- optional `configuration`
- optional `local_folder_path`
- optional `pipeline_backup_directory`

It manages:

- registered blocks
- `para_value_dict`
- artifact registry
- run history
- metadata directory
- logger
- optional backup metadata used by backup-aware save/load

If `configuration` is omitted, the pipeline starts with an empty config.

If `local_folder_path` is omitted, the pipeline automatically uses a temporary staging root. This is useful when building a child pipeline in a notebook before attaching it to a parent. Once attached, the child moves under the parent project tree and the temporary root is no longer its working location.

This staging root is a notebook-friendly convenience:

- unattached temporary-root pipelines are cleaned up during normal object or runtime cleanup
- if such a pipeline is attached to a parent, it is relocated under the parent tree
- if such a pipeline is saved to an explicit path, its `project_root` is materialised in that directory
- for a durable standalone location from the beginning, pass an explicit `local_folder_path`

### 2. ExecutionBlock

Each block has:

- a block name
- a numeric execution priority
- one or more registered functions

Independent functions registered in the same block run in parallel. Blocks remain explicit execution boundaries and run according to priority.

Parent-level execution can also use float priorities for branch groups. For example, `5.1`, `5.3`, and `5.9` all belong to group `5`. Once one node in that integer-priority group actually executes, later nodes in the same group are skipped automatically.

### 3. Argument resolution

When executing a registered function, inputs are resolved in this order:

1. explicit runtime overrides
2. `para_value_dict`
3. config fields
4. function defaults

Special cases:

- if a function declares an argument named `logger`, the pipeline logger is injected automatically
- child pipelines can use upstream parent outputs from earlier parent-level nodes
- child config values override same-named parent config values
- child config values are not exposed to parent blocks

### 4. Outputs

Function outputs can be:

- stored directly in memory
- stored on disk if listed in `save_to_disk`

Disk-backed outputs are represented in memory by `ArtifactRecord`, but can be resolved back to real values using `get_value(...)`.

Current serialisation behaviour:

- JSON-serialisable values use `json`
- `numpy.ndarray` values use `.npy`
- PyTorch tensors and modules use `torch`
- pandas DataFrames use Feather by default
- pandas DataFrames with more than 3 million rows use a single Parquet file
- Dask DataFrames use Parquet directories and remain Dask on reload
- everything else falls back to `pickle`

### 5. Rename function inputs and use variadics safely

When a function uses generic names like `obj`, or uses `*args` / `**kwargs`, you can expose safer pipeline-facing names during registration.

```python
def mapped_variadic(obj: int, *more_values: int, scale: int = 1, **extra_values: int) -> int:
    return (obj + sum(more_values) + sum(extra_values.values())) * scale


block.register_function(
    mapped_variadic,
    ["result"],
    param_mapping={"obj": "payload", "scale": "scale_value"},
    var_pos_name="extra_args",
    var_kw_name="extra_kwargs",
)
```

This lets the pipeline resolve:

- `payload` → original `obj`
- `scale_value` → original `scale`
- `extra_args` → original `*more_values`
- `extra_kwargs` → original `**extra_values`

Rules:

- renamed variadic positional values must resolve to a `list` or `tuple`
- renamed variadic keyword values must resolve to a `dict`
- mapping metadata is preserved on save/load
- if the same function is already registered in a block, use `forced=True` to replace it

### 6. Run modes

Available execution methods:

- `run_all()`
- `run_until(block_name)`
- `run_from(block_name)`
- `run_block(block_name)`

Nested targeting is also supported by path:

```python
pipeline.run_until("modeling_pipeline", "predictor_components")
pipeline.run_from("modeling_pipeline", "predictor_training_pipeline", "predictor_saving_pipeline")
pipeline.run_block("modeling_pipeline", "predictor_components")
```

### 7. Manage configuration

```python
pipeline.set_config({"multiplier": 10})
```

Rules:

- the pipeline may be created with `configuration=None`, which is treated as an empty config
- `set_config(...)` adds new fields or updates existing ones
- `update_config(...)` updates existing fields only
- config writes that would conflict with declared output names are rejected or skipped depending on the method used

```python
full_config = pipeline.get_full_config()
model_cls = pipeline.get_config_value("model_cls")
```

Behaviour:

- `get_full_config()` returns the visible merged config for the pipeline
- parent configs are included recursively for nested child pipelines
- current pipeline config overrides same-named parent values
- `get_config_value(name)` raises if the key does not exist
- child configs are not propagated upward to parents or siblings

### 8. Access values safely

```python
value = pipeline.get_value("model_blob")
```

If the value is disk-backed, the true object is loaded and returned.

To modify values:

```python
pipeline.update_value("existing_name", 10)
pipeline.set_value("new_or_existing_name", 20)
```

Behaviour:

- `update_value(...)` updates existing visible values only
- `set_value(...)` creates a new pipeline-owned value if it does not exist, otherwise it updates the existing value
- values created with `set_value(...)` are visible to the pipeline, its descendants, and downstream siblings through the parent visibility model

### 9. Save and load projects

```python
pipeline.save_pipeline()
loaded = PipelineHandler.load_pipeline("demo_run")
```

Without a path, `save_pipeline()` saves to the current `project_root`.

Passing a different path creates a restart-safe copy of the current project tree before writing fresh state and metadata. The copy includes nested child directories and disk-backed artifacts. The target must not overlap the current project tree.

You can also configure a backup directory when creating the pipeline:

```python
pipeline = PipelineHandler(
    "demo",
    config,
    Path("demo_run"),
    pipeline_backup_directory=Path("demo_backup"),
)
```

Save and load behaviour:

- if `pipeline_backup_directory is None`, an in-place save updates only the working tree
- if `pipeline_backup_directory` is set, an in-place save also refreshes the backup copy
- `load_pipeline(path)` reads lightweight metadata before loading the full state
- if `path` is the canonical working directory, the pipeline loads directly
- if `path` differs from the canonical working directory, the library restores the saved tree to the canonical directory before loading it
- `load_pipeline(path, forced_deleting=False)` asks for keyboard confirmation with `yes` or `y` before deleting a non-empty canonical working directory during restore
- `load_pipeline(path, forced_deleting=True)` deletes the canonical working directory directly during restore

Saved projects contain:

- `config.pkl`
- `pipeline_state.pkl`
- disk-backed outputs under `artifacts/`
- logs and configuration snapshots under `metadata/`

Saved pipelines preserve callable references rather than historical source code. Importable callables are restored from their import paths. Notebook-local functions and other runtime-only callables must be defined or imported under the same name before calling `load_pipeline(...)`. A saved project copies pipeline data, not Python source, installed packages, or the runtime environment.

Compatibility aliases `save_project()` and `load_project()` still exist.

### 10. Print the pipeline chart

```python
print(pipeline.describe_pipeline())
print(pipeline)
print(repr(pipeline))
```

Current chart format includes:

- block name
- priority
- child pipeline hierarchy
- gate block
- function name
- only argument names that are actually supplied by visible configs or earlier outputs
- output names
- `*` marker for disk-backed outputs

Additional chart behaviour:

- child pipelines gated off by config are greyed out when the current config value does not match the gate’s expected value
- the root pipeline is never greyed out this way

Gate lines do not show `-> bool`, and chart symbols such as `()` and `->` use the same colour family as priority markers for readability.

### 11. Output conflicts and overrides

Duplicate output names across different parent-level blocks or child pipelines are allowed.

- later parent-level nodes override earlier parent-level nodes
- child internal override chains are not expanded in the parent conflict report

Helpers:

```python
conflicts = pipeline.get_output_conflicts()
print(pipeline.describe_output_conflicts())
```

### 12. Logging

The pipeline creates a logger automatically.

Supported methods:

- `debug(...)`
- `info(...)`
- `warning(...)`
- `error(...)`
- `critical(...)`
- `result(...)`
- `print(...)`

Behaviour:

- every log line includes a UTC timestamp
- all log lines are appended to `metadata/pipeline.log`
- `result(...)` messages are kept in a separate in-memory history list
- `print(...)` inside registered functions can also be captured into the logger

Logger helpers:

```python
history = pipeline.get_result_history()
pipeline.print_result_history()
pipeline.clear_result_history()
pipeline.set_print_capture_mode("tee")

# Optional: pause and resume file logging.
pipeline.logger.disable_file_logging()
pipeline.logger.enable_file_logging()
```

`clear_result_history()` only clears in-memory result history. It does not modify `metadata/pipeline.log`.

Normal notebook use does not require manual logger cleanup. `disable_file_logging()` pauses writes to `metadata/pipeline.log` while console output and in-memory result history continue. `enable_file_logging()` resumes appending without clearing existing entries. Attached children share their parent logger and do not manage its lifetime.

Print capture modes:

- `tee` (default): send `print(...)` output to both normal stdout and the pipeline log
- `logger_only`: capture `print(...)` output only into the pipeline log
- `off`: leave normal `print(...)` behaviour unchanged

### 13. Memory options

You can enable optional runtime memory tools when creating a pipeline:

```python
pipeline = PipelineHandler(
    ...,
    memory_saving_mode=True,
    memory_profile_logging=True,
)
```

Behaviour:

- `memory_saving_mode=True` runs a best-effort cleanup after each block finishes
- `memory_profile_logging=True` logs memory after computation and cleanup for each block
- attached child pipelines inherit these settings from their parent

## Example notebook

Open and run:

```text
examples/comprehensive_pipeline.ipynb
```

The notebook writes its runtime data under:

- `examples/example_run/`

