Pipeline Docs

Synthetic Clinical Data Generator

Welcome! This tool allows you to generate fake patient data. Instead of hard-coding values, this system uses a flexible "blueprint" (a YAML configuration file) to build a relational database of patients, vitals, labs, and medications.

💡 For Dummies: Think of this tool as a factory. The data_config.yaml file is the recipe book that tells the factory what to build. The Python scripts are the machines that actually build it.

1. Global Parameters (The "Big Picture")

At the very top of your data_config.yaml file, you will find the generation_params. This dictates the size and scope of your fake hospital.

generation_params:
    n_patients: 100         # How many fake patients to create
    days: 10                # How many days of data to simulate
    freq: '4h'              # How often to check vitals (every 4 hours)
    output_format: 'tidy'   # The shape of the final data
    default_missing_rate: 0.5 # 50% chance a test isn't taken (makes data realistic)

2. The Clinical Dictionary (The "Menu")

Before building tables, we define universal medical concepts under clinical_concepts. This ensures the generator never creates impossible scenarios (like a body temperature of 100°C) and standardizes coding.

Concepts are grouped into vitals, labs, and neuro. Each concept defines:

3. Table Architectures

⚙️ Under the Hood: The tables section dictates how CSVs are structured and how they relate. Let's break down the core architectural rules.

A. Table Types & Row Counts

Every table must define its basic structural behavior:

B. The `map_to` Keyword (The Translator)

In eav_timeseries tables (like vitals or lab results), the generator produces raw data first. You use map_to to tell the generator exactly which column in your final CSV should hold which piece of data.

OBSERVATION_CODE:
  map_to: "concept.code"  # Takes the LOINC code from the dictionary
OBSERVATION_NAME:
  map_to: "concept.name"  # Takes the human-readable name
OBSERVATION_RESULT_CLEAN:
  map_to: "value"         # Takes the actual randomly generated number (e.g., 98.6)

4. Column Types Deep-Dive

Inside the schema of a table, you define your columns. Here is exactly what every column configuration does, with examples:

1. The Combo Meal: categorical_tuple

This is crucial for clinical accuracy. If a patient has diabetes, their diagnosis code should always match the description. If you generated them separately, you might accidentally give a patient an Asthma code with a Diabetes description. A categorical_tuple locks them together.

PROBLEM_TUPLE:
  type: "categorical_tuple"
  columns: [ "PROBLEM_CODE", "PROBLEM_DESC" ] # The columns to create
  values:
    # If the generator picks line 1, it inserts both "E11.9" and the Diabetes text safely.
    - [ "E11.9", "Type 2 diabetes mellitus" ]
    - [ "I10", "Essential (primary) hypertension" ]
    - [ "J44.9", "Chronic obstructive pulmonary disease" ]

2. Time Travel Prevention: date and date_offset

Medical data must follow a timeline. You can't resolve a medical problem before you've diagnosed it. The generator uses offsets to enforce this logic.

PROBLEM_DT_TM:
  type: "date"
  start: "2020-01-01"
  end: "2024-01-01"  # Picks a random baseline date in this window.

UPDATE_DT_TM:
  type: "date_offset"
  base_col: "PROBLEM_DT_TM" # Look at the problem date we just generated...
  days_range: [ 0, 90 ]     # ...and add anywhere from 0 to 90 days to it.

Result: If the problem started on Jan 1st, the update date is mathematically guaranteed to happen between Jan 1st and April 1st. No time paradoxes!

3. The V.I.P. Pass: foreign_key

This is how tables talk to each other. If you generate a prescription, it needs to belong to a valid hospital visit.

ENCNTR_ID:
  type: "foreign_key"
  source_table: "ICARE_EPISODES_ANON"

This tells the script: "Go look at the Episodes table, find a valid Encounter ID for this specific patient, and paste it here."

4. Simple Generators

5. The Configured Tables

The data_config.yaml currently builds 6 interconnected tables for your cohort:

# Table Name Description
1 ICARE_EPISODES_ANON The anchor table. Records admissions, discharges, age, and deprivation deciles.
2 ICARE_MICROBIOLOGY_ANON Tracks blood/urine cultures, organism growth (e.g., E. coli, MRSA), and sensitivities.
3 ICARE_VITAL_SIGNS_ANON Dense time-series of heart rate, temperature, blood pressure, etc., tracked longitudinally.
4 ICARE_PROBLEMS_ANON ICD-10 diagnostic history (diabetes, sepsis, CKD) with onset and resolution dates.
5 ICARE_PHARMACY_PRESCRIBING_ANON Medication orders (antibiotics, pressors, fluids) with routes and dosages.
6 ICARE_PATHOLOGY_BLOOD_ANON Lab test results (creatinine, CRP, lactate) featuring delayed result timestamps.

6. How Data is Created

The Python backend (src/generators.py) executes the YAML blueprint sequentially. It establishes parent tables (Episodes) first so child tables (Pharmacy, Vitals) can safely inherit foreign keys. It then applies biological variance, bounds logic, and missingness masks to produce highly realistic datasets.

7. Running the Tool

Use the unified orchestration suite to execute data generation. This will automatically route through Docker (or run locally if specified) and output your CSVs to a timestamped folder in data/synthetic/.

# Default (Runs in Docker Container)
make generate

# Native Execution (Runs on Host CPU)
make generate local

Note for Windows users: Use .\make.bat generate.

On this page

1. Global Parameters 2. The Clinical Dictionary 3. Table Architectures 4. Column Types Deep-Dive 5. The Configured Tables 6. How Data is Created 7. Running the Tool