Model Evaluation & Stratification

Welcome to Step 3 of the pipeline! You have generated raw hospital data and engineered complex clinical features. Now, we must ask the most important question: Are these clinical scores actually accurate?

💡 The "God Config" Concept: In the past, changing an experiment required rewriting Python code. Now, this pipeline is 100% configuration-driven. The Python script acts as a "Dumb Dispatcher." It simply reads your instructions from eval_config.yaml and builds the exact experiment you ask for.

Let's walk through the four sections of your eval_config.yaml Mission Control panel.

1. Global Settings (The Target)

This section defines the core rules of the experiment. What are we trying to predict, and how are we measuring success?

experiment:
  target_label: 'ground_truth'
  scores_to_evaluate: 'all'
  metrics_to_compute: 'all'

target_label: The exact name of the column containing the true outcome (e.g., sepsis_case or ground_truth).
scores_to_evaluate: You can type 'all' to automatically evaluate every custom score you built, or pass a specific list to isolate tests: ['mews_score', 'pitt_score'].
metrics_to_compute: The specific math you want printed to your CSV summaries (AUROC, Sensitivity, PPV, etc.).

2. Time-Slice Strategies

A patient's health changes drastically from Day 1 to Day 5. Therefore, a clinical score's accuracy also changes! We evaluate the scores at specific "snapshots" in time.

time_slices:
  admission:
    run: true
    plots_to_generate: ['roc_curve', 'pr_curve']
  milestones:
    run: true
    hours: [24, 48, 72]

Admission: Grabs the very first recorded row for the patient. (Best for evaluating Triage tools like MEWS).
Peak: Grabs the absolute highest (worst) score recorded during their entire stay. (Best for retrospective severity scales like Pitt Bacteremia).
Milestones: Calculates the exact hours since admission and evaluates the score strictly at the 24h, 48h, or 72h mark.

⚠️ Continuous Mode Warning: Setting continuous: true evaluates every single 4-hour interval independently. While useful for debugging, it will artificially inflate your accuracy metrics because one sick patient with 30 rows counts as 30 "correct" predictions!

3. Longitudinal Analyses (Performance Over Time)

Instead of single snapshots, what if we want to see a line graph showing how the AUROC degrades or improves over an entire week?

longitudinal:
  auroc_over_time:
    run: true
    bin_hours: 24
    max_hours: 168

If set to true, the pipeline chops the patient's stay into 24-hour buckets, calculates the AUROC for each day, and draws a beautiful line graph up to 168 hours (7 days).

4. Subgroup Stratification (The Multi-Filter Engine)

A score might look 85% accurate overall, but what if it is only 50% accurate for elderly patients? To ensure algorithmic fairness, we define Cohorts using a list of filters.

⚙️ How it Works: The Python engine reads your filters and translates string operators (like ">=") into actual math (like operator.ge) to slice the dataset before evaluating.

subgroup_analysis:
  run: true
  cohorts:
    - name: "Elderly Diabetics"
      filters:
        # Filter 1: Must be older than 65
        - column: "AGE_AT_ADMISSION"
          operator: ">="
          value: 65
        # AND Filter 2: Must have diabetes
        - column: "hx_diabetes_uncomp"
          operator: "=="
          value: 1

When you run the script, it will take the "Elderly Diabetics" sub-population and pass them through every active Time-Slice strategy (Admission, Peak, Milestones) independently!

5. Running the Script & Understanding Outputs

Once your YAML is configured, simply run:

python -m scripts.03_evaluate_scores

Where do the files go?

The src/metrics.py engine automatically sanitizes your cohort names (turning "Elderly (>= 65)" into "elderly_65") and saves everything into an organized outputs/ folder.

📦 project_root/
 ┗ 📂 outputs/
   ┣ 📂 metrics/
   ┃ ┣ 📜 master_summary.csv                  <-- All scores, all cohorts, in one file!
   ┃ ┣ 📜 all_patients_admission_metrics.csv
   ┃ ┗ 📜 elderly_diabetics_peak_metrics.csv
   ┗ 📂 plots/
     ┣ 🖼️ all_patients_admission_roc.png
     ┣ 🖼️ elderly_diabetics_peak_pr.png
     ┗ 🖼️ temporal_auroc.png                  <-- Your longitudinal line graph

🎉 Master Summary: The master_summary.csv is your golden ticket. You can open it in Excel, filter by the "Strategy" column, and instantly compare how the MEWS score performs for Young vs. Elderly patients on Day 1!

Pipeline Docs