Pipeline Docs

Feature & Phenotype Builder

Welcome to the Feature Pipeline! Raw hospital data is incredibly messy—patients miss lab tests, doctors use different diagnosis codes, and vitals are recorded at random times. Before we can build predictive AI models, we must translate this raw data into clean, mathematical signals.

💡 For Dummies: Think of feature_config.yaml as the control panel. Instead of writing hundreds of lines of complex Python code to clean data and search for ICD-10 codes, you simply declare what you want in this file. The background Python scripts act as an automated factory that reads your rules and builds the final dataset.

The pipeline operates in four strict phases, moving from simple cleaning to complex medical algorithms.

Phase 1: Base Features (Time-Series Cleaning)

This section of the YAML file handles raw numbers that change over time (like heart rate, blood pressure, or temperature). It fixes missing data and calculates trends automatically using the logic inside features.py.

base_features:
  hr:
    impute: 'ffill'
    missing_indicator: false
    delta: true
    rolling:
      windows: ['24h', '7D']
      aggs: ['mean', 'max']

Exhaustive Parameter Breakdown:

Phase 2: Computed Features (Fast Math)

Once the base features are clean, we can perform basic math on them. This section uses the Pandas df.eval() engine, which allows you to write simple math equations as plain text strings.

computed_features:
  # Simple division for clinical ratios
  shock_index: "hr / sbp"

  # Boolean Logic (Returns 1 for True, 0 for False)
  qsofa_score: "(rr >= 22) + (sbp <= 100)"
🧮 How Boolean Math Works: In Python, True equals 1 and False equals 0. By putting equations in parentheses and adding them together, you can instantly build simple scores. In the qsofa_score example, if a patient has a Respiratory Rate of 25 (True=1) and a Systolic BP of 120 (False=0), their score automatically evaluates to 1.

Phase 3: Custom Features (Phenotypes)

A "Phenotype" is a clinical state. We cannot use simple math to find out if a patient has "Metastatic Cancer" or "Diabetes". We have to search through text, pharmacy records, and billing codes (ICD-10).

custom_features:
  hx_mi: # This will be the name of the new column (History of Myocardial Infarction)
    module: 'src.phenotypes'
    function: 'derive_historical_condition'
    kwargs:
      target_codes: ['I21', 'I22', 'I25.2', '323..00', 'G30..00']

How It Works:

  1. module & function: Tells the system exactly which Python script and which function to run. In this case, it runs the derive_historical_condition function.
  2. kwargs (Keyword Arguments): This is where you pass specific instructions to the Python function. The function will dig through the patient's entire medical history (like the ICARE_PROBLEMS_ANON table) and, if it finds any of those codes using prefix matching, it will put a 1 in the hx_mi column.
⚙️ Complex Example: Temporal Pharmacy Search
Look at the has_vasopressors rule in the YAML. It calls has_medication_in_window and passes a window_hours: 24 and a list of target_meds. The Python code behind this uses the _get_prescriptions_in_window helper to temporally align the patient's vitals with the pharmacy database, check if any of the drugs (like 'epinephrine') were administered within exactly 24 hours of that specific moment in time, and return a 1 or 0.

Phase 4: Final Clinical Scores

This is the top of the pyramid. Now that we have clean vitals, computed math, and complex 1/0 phenotypes, we can calculate validated medical scores (like Charlson Comorbidity Index or INCREMENT-ESBL).

custom_scores:
  charlson_quan_score:
    module: 'src.scores'
    function: 'calculate_charlson_quan'
    kwargs:
      age_col: 'AGE_AT_ADMISSION'
      mi_col: 'hx_mi'
      chf_col: 'hx_chf'
      # ... other mappings

The Golden Rule: The kwargs here map the names of the columns you generated in Phase 3 to the variables expected by the medical calculator. It says: "Hey Calculator, when you need to know if the patient has a history of MI, look inside the column named hx_mi."

5. Tutorial: Adding a New Feature (Acute Kidney Injury)

Let's walk through an exhaustive example. We want to create a new phenotype called Acute Kidney Injury (AKI). A patient is flagged for AKI if they have the ICD-10 code "N17" OR if their rolling Creatinine lab test is severely elevated (e.g., > 1.5 mg/dL).

Step 1: Update feature_config.yaml

First, we need to make sure Creatinine is being processed as a base feature, and then we declare our new custom phenotype.

# 1. Add Creatinine to Base Features so we get rolling windows
base_features:
  creatinine:
    impute: 'ffill'
    rolling:
      windows: ['24h']
      aggs: ['max']

# 2. Add the new AKI phenotype to Custom Features
custom_features:
  is_aki:  # This will be the name of the new column
    module: 'src.phenotypes'
    function: 'derive_aki_status'
    kwargs:
      creatinine_col: 'creatinine_24h_max'
      target_codes: ['N17']

Step 2: Write the Logic in src/phenotypes.py

Next, we open phenotypes.py and write the derive_aki_status function. We will use the built-in helper functions to keep the code clean and fast.

def derive_aki_status(df, **kwargs):
    """
    Derives Acute Kidney Injury (1=Yes, 0=No).
    Logic: ICD-10 code 'N17' OR Creatinine > 1.5.
    """
    # Start by assuming no AKI (fill column with 0s)
    flag = pd.Series(0, index=df.index)

    # Rule 1: Check for Historical ICD-10 Codes
    target_codes = kwargs.get('target_codes', [])
    has_code = _patient_has_historical_codes(
        df=df,
        context_df=kwargs.get('context_dfs', {}).get('problems'),
        patient_col='SUBJECT',
        code_col='PROBLEM_CODE',
        target_codes=target_codes
    )
    flag.loc[has_code] = 1

    # Rule 2: Check the rolling Lab Value Proxy
    creat_col = kwargs.get('creatinine_col')
    if creat_col in df.columns:
        creat = pd.to_numeric(df[creat_col], errors='coerce')
        flag.loc[creat > 1.5] = 1

    return flag.values

Step 3: What the Output Looks Like

When you run the pipeline, the system will automatically pull the data, apply the rolling window to creatinine, execute your new function, and append the is_aki column to your final dataset.

patient_id date creatinine creatinine_24h_max is_aki Explanation
101 2024-01-01 08:00 0.9 0.9 0 Healthy patient, normal creatinine, no ICD codes.
102 2024-01-01 12:00 1.6 1.6 1 Flagged! Creatinine is > 1.5.
103 2024-01-02 08:00 1.1 1.1 1 Flagged! Labs are normal, but patient has the 'N17' ICD-10 code in their history.

6. Under the Hood: The Python Backend Rules

If you are a developer looking to add new medical rules, you must adhere to these strict naming conventions and architectural rules:

1. phenotypes.py (The Translation Layer)

This file extracts messy data and turns it into clean 1s and 0s. Do not put final predictive score math here.

2. scores.py (The Rule Engine)

This file only does math. Do not search for ICD-10 codes here.

This file relies heavily on a custom function called evaluate_score. It allows you to build a transparent "rulebook" for a clinical score. For example, the INCREMENT-ESBL score is defined simply as a list of rules:

rules = [
    {'desc': 'Age > 50', 'col': age_col, 'condition': df[age_col] > 50, 'points': 3},
    {'desc': 'Charlson > 3', 'col': charlson_col, 'condition': df[charlson_col] > 3, 'points': 4},
]
return evaluate_score(df, rules, score_name, verbose, logger)
✅ Why we use this rule engine: Because clinical scores require auditing! If a doctor asks "Why did my patient get an INCREMENT score of 7?", the evaluate_score engine automatically prints a clean trace (e.g., [+] Age > 50 (Value: 72) +3 Points), making your AI pipeline 100% transparent and explainable.

On this page

0. Pipeline Overview 1. Phase 1: Base Features 2. Phase 2: Computed Features 3. Phase 3: Custom Phenotypes 4. Phase 4: Clinical Scores 5. Tutorial: Adding a New Feature 6. Python Backend Logic