Pipeline Architecture & Orchestration

This document outlines the consolidated folder structure of the DM-ESBL clinical validation pipeline and explains how to utilize the unified cross-platform automation suite to execute workflows seamlessly.

Environment Initialization

Before running any pipeline steps, initialize the isolated container infrastructure. This ensures all dependencies, libraries, and python environments match production standards exactly, eliminating cross-platform configuration errors.

# Clone, build, and launch the isolated pipeline container network in detached mod
 docker-compose up -d --build

Once the container status is active, you can utilize the orchestration suite below to run commands directly inside the containerized system environment.

1. Folder Structure

The project strictly separates configuration blueprints, core modular logic, execution dispatchers, and generated artifacts to ensure complete auditability, reproducibility, and isolation of experimental setups.

dm-esbl/
├── assets/                 # Static clinical knowledge bases and maps
│   └── res195-comorbidity-cci-gold.csv
├── config/                 # Blueprints controlling pipeline execution (YAML)
|   ├── code_search.yaml     # Keywords to look for insteresting codes
│   ├── data_config.yaml     # Synthetic data generation schemas
│   ├── feature_config.yaml  # Feature windowing and phenotype logic definitions
│   ├── eval_config.yaml     # Cohort stratification and time-slice rules
│   └── threshold_config.yaml# Literature-recommended safety validation cutoffs
├── data/                   # Data storage layer (Ignored by Git)
│   ├── synthetic/            # Unprocessed multi-table synthetic cohorts (Step 1 output)
│   └── processed/            # High-dimensional patient-level analytics (Step 2 output)
├── outputs/                # Target summary matrices and evaluation plots
│   └── <run_timestamp>/     # Isolated, directory-contained results per experiment run
│       ├── metrics/        # Tabular performance reports (Master summary, stewardship, etc.)
│       └── plots/          # High-resolution clinical curves (ROC, PR, Longitudinal)
├── reports/                # Validation logging and static audit trails
│   ├── score_validation.log
│   └── code_search_results.txt
├── scripts/                # Executable orchestrators and pipeline dispatchers
│   ├── 01_generate_data_v2.py
│   ├── 02_build_features_icare.py
│   ├── 03_evaluate_scores_v2.py
│   ├── 04_evaluate_thresholds.py
│   ├── 05_validate_scores.py
│   └── 06_find_clinical_codes.py
├── src/                    # Production core logic assets (Pure mathematical functions)
│   ├── generators.py        # Simulation and EAV cohort generation algorithms
│   ├── features.py          # Aggregation engine, windowing, and parsing systems
│   ├── phenotypes.py        # Clinical rule extraction and context matching logic
│   ├── scores.py            # Transparent clinical point calculators with audit traces
│   ├── metrics.py           # Statistical validation, curves, and evaluation engines
│   └── utils.py             # Multi-stage execution helper modules
├── tests/                  # Regression testing suite checking for logic drift
│   ├── cases.csv            # The "Gold Standard" human-verified patient audits
│   ├── test_phenotypes.py
│   └── test_scores.py
├── Dockerfile               # Pipeline container runtime instructions
├── docker-compose.yml       # Shared infrastructure orchestration mount
├── Makefile                 # Linux / macOS Pipeline automation suite
└── make.bat                 # Windows Unified Pipeline automation proxy

2. The Unified Orchestration Suite

The system provides dual managers (Makefile and make.bat) featuring an identical syntax wrapper. This design decouples runtime environments from physical machine operating systems, allowing you to control execution using the exact same terminology whether on Linux or Windows host targets.

📋 CLI Target Overrides: Running any command with no target (simply typing make or .\make.bat) automatically triggers the internal help manual displaying available automation hooks.

Unix Target	Windows Command	Action Scope
`make all`	`.\make.bat all`	Runs end-to-end simulation, compilation, feature extraction, mathematical profiling, safety thresholds, and validation (Steps 1-5).
`make generate`	`.\make.bat generate`	Step 1: Spins up structural synthetic relational clinical cohorts.
`make features`	`.\make.bat features`	Step 2: Cleans records, computes windows, extracts phenotypes, and logs scores.
`make evaluate`	`.\make.bat evaluate`	Step 3: Quantifies risk scores via configuration-driven stratification metrics.
`make thresholds`	`.\make.bat thresholds`	Step 4: Conducts literature safety checks focused on maximizing NPV.
`make validate`	`.\make.bat validate`	Step 5: Pushes test patients from case manifests into code verification checks.
`make search`	`.\make.bat search`	Utility: Performs query extraction for coding classifications.
`make test`	`.\make.bat test`	Utility: Validates functional assertions inside testing environments using Pytest.
`make clean`	`.\make.bat clean`	Utility: Wipes out stale verification logs and reporting exports.

3. Runtime Environments

The managers leverage a runtime routing mechanism designed to adapt configuration based on execution goals.

A. Containerized Environment (Default Workflow)

By default, executing automation targets directly routes code execution into the isolated pipeline container. This enforces isolation of software environments, paths, and package models across target environments.

# Initial boot setup
docker-compose up -d --build

# Subsequent execution targets process automatically within the active container infrastructure
make generate
make features

B. Native Host Environments (The Local Toggle)

To run directly inside local development terminal settings (bypassing container runtimes), append the explicit keyword indicator local to any automation command. This switches execution directly back to local machine Python configurations.

# Execution utilizes localized paths, dependencies, and resources
make generate local
.\make.bat features local

4. Parameterized Argument Injection

All core pipeline scripts are constructed using modular configuration parsers. You can pass explicit runtime flag modifications downstream into execution layers without restructuring hardcoded YAML blueprints or script files.

Execution Formatting Patterns

Unix Systems (via the ARGS environment variable pass-through block):

make generate ARGS="--config experimental_data_blueprint.yaml"
make evaluate local ARGS="--eval-config strict_sepsis_matrix.yaml"

Windows Platforms (via sequential string argument chain wrapping):

.\make.bat generate --config experimental_data_blueprint.yaml
.\make.bat evaluate local --eval-config strict_sepsis_matrix.yaml

Pipeline Docs