Pipeline Validation & Quality Assurance

In clinical machine learning, a single logic error can have cascading effects. If a function mistakenly categorizes a patient's temperature or misreads a medication history, the final risk score will be entirely compromised. This is why checking that each individual step is working fine is the most critical part of your pipeline.

⚠️ The "Garbage In, Garbage Out" Rule: A predictive model is only as intelligent as the features it is trained on. Before you evaluate a model's AUROC, you must mathematically guarantee that the underlying clinical scores are calculating points correctly.

2. The Core Components to Check

Your data transforms through three major stages. We strongly encourage you to manually review and periodically test the functions within each of these files:

features.py: Are your rolling windows accurately capturing the "24-hour maximum" heart rate? Are missing values being imputed correctly?
phenotypes.py: Are the ICD-10 regex searches correctly flagging historical conditions? Is the pipeline accurately converting raw medication strings into clean 1 (True) or 0 (False) flags?
scores.py: Are the clinical scoring algorithms (like INCREMENT-ESBL or MEWS) adding up the points perfectly based on the phenotype flags?

3. Validating Clinical Scores

To ensure our final scores are mathematically flawless, we use a dedicated Unit Testing approach. We have created two specific files for this purpose: cases.csv and 05_validate_scores.py.

How it works: The cases.csv file contains synthetic "Test Patients". For each patient, we have manually calculated what their clinical scores should be (e.g., exp_increment, exp_gavaghan). The validation script then runs these patients through your Python code and checks if the computer's math matches the human's math.

Execution

To run the validation test, execute the following command from your project root:

python -m scripts.05_validate_scores

4. Reading the Audit Report

When you run the validation script, it generates a highly detailed audit log located at reports/score_validation.log. This report acts as a clinical receipt, showing you exactly how the script arrived at its final number.

What to look for: Open the report and look for the ✅ MATCH indicators. If you see a ❌ MISMATCH, the report will show you exactly which patient and which score failed, allowing you to debug the specific function in scores.py.

▶ CASE ID: 1 | High-Risk Tertiary
============================================================
  [INCREMENT-ESBL Breakdown]
    [+] Age > 50                   (Value: 60)   +3
    [+] Charlson > 3               (Value: 5)    +4
    [+] Pitt Score >= 6            (Value: 7)    +3
    [+] SIRS >= 2                  (Value: 3)    +4
    [+] Non-Urinary Source         (Value: 1)    +3
    [+] Non-E. coli                (Value: 1)    +2
    [+] Inappropriate Abx          (Value: 1)    +2
    ---------------------------------------------
    [=] TOTAL COMPUTED:            21

    ✅ MATCH: Expected 21, got 21

5. Creating New Unit Tests

As you add new clinical phenotypes or scoring systems to the pipeline, you must accompany them with unit tests. This prevents "logic drift" where a small change in one function accidentally breaks a clinical threshold elsewhere.

The Two Levels of Testing

Test Type	Target File	Purpose
Phenotype Test	`test_phenotypes.py`	Verify binary logic (e.g., Is 91 bpm actually > 90?).
Score Test	`test_scores.py`	Verify point summation and hierarchy (e.g., Does Metastatic Cancer trump Solid Tumor?).

Step-by-Step Example: Testing a New Vital Sign

If you implement a new function is_hypotensive() in phenotypes.py, follow these steps to validate it:

Step 1: Define the Mock Scenario

Create a small DataFrame with "Edge Cases" (values exactly at the threshold). For a SBP threshold of 90 mmHg, test 89, 90, and 91.

def test_is_hypotensive_logic():
    # Setup data with edge cases
    data = pd.DataFrame({
        'sbp': [89, 90, 120]
    })

    # Run the phenotype function
    # Expected: 89 is True (1), 90 is False (0), 120 is False (0)
    result = is_hypotensive(data, sbp_col='sbp')

    assert result.iloc[0] == 1  # 89 mmHg
    assert result.iloc[1] == 0  # 90 mmHg (Threshold is strictly < 90)
    assert result.iloc[2] == 0  # 120 mmHg

Step 2: Account for Keyword Arguments (kwargs)

Clinical functions often use **kwargs to find column names. Ensure your test explicitly passes the column names used in your mock DataFrame so the function doesn't default to production names (like sbp_24h_min) and fail to find your data.

Pro Tip: When testing scores in test_scores.py, use the score_configs fixture. This ensures your test uses the same clinical weights defined in config/feature_config.yaml.

Best Practices for Clinical Validation

Test the "Min" and "Max": Always include a test case where a patient has 0 points and one where they have the absolute maximum possible points.
Handle Overlaps: If a patient triggers two rules for the same category (e.g., both high and low temperature), ensure your function uses a logical OR (|) so it returns 1 (True), not 2.
Return Series: Ensure your phenotype functions return a pd.Series rather than a numpy.ndarray so you can use .iloc and .loc in your assertions.

Pipeline Docs