Metadata-Version: 2.4
Name: ftir-prep
Version: 0.3.2
Summary: A framework for designing and evaluating optimal preprocessing pipelines for FTIR spectral data used in classification tasks. It provides modular implementations of common preprocessing techniques and allows automated exploration of preprocessing combinations to enhance model performance.
Home-page: https://github.com/username/ftir-preprocessing-framework
Author: Lucas Mendonça
Author-email: Lucas Mendonça <lucas.mendonca@example.com>
Maintainer-email: Lucas Mendonça <lucas.mendonca@example.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/lucas-mendonca-andrade/FTIR-Prep
Project-URL: Repository, https://github.com/lucas-mendonca-andrade/FTIR-Prep
Project-URL: Bug Tracker, https://github.com/lucas-mendonca-andrade/FTIR-Prep/issues
Project-URL: Documentation, https://github.com/lucas-mendonca-andrade/FTIR-Prep/docs
Keywords: ftir,spectroscopy,preprocessing,machine-learning,optimization,bioinformatics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Chemistry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: optuna>=3.0.0
Requires-Dist: rampy>=0.1.0
Requires-Dist: PyWavelets>=1.9.0
Requires-Dist: statsmodels>=0.13.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: shap>=0.41.0
Requires-Dist: openpyxl>=3.0.0
Requires-Dist: joblib>=1.2.0
Requires-Dist: psutil>=5.9.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: flake8>=4.0; extra == "dev"
Requires-Dist: mypy>=0.950; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
Requires-Dist: myst-parser>=0.17; extra == "docs"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# FTIR-Prep: FTIR Preprocessing Framework

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Version](https://img.shields.io/badge/version-0.3.1-orange.svg)](https://github.com/username/ftir-preprocessing-framework)

A modular and extensible framework for optimizing FTIR preprocessing pipelines for disease diagnosis.

## 🚀 Features

- **Modular**: Component-based reusable architecture
- **Extensible**: Easy addition of new preprocessing techniques
- **Robust Pipeline Search**: `RobustPipelineSearch` finds a preprocessing pipeline that performs consistently across a pool of classifier families (not just tuned to one), with time/trial and memory budgets, parallelization, and checkpointing to disk
- **Robust Validation**: Support for group-based cross-validation
- **Configurable**: Flexible pipeline configuration system
- **Documented**: Complete documentation with practical examples

## 📋 Supported Preprocessing Techniques

### 🔧 Baseline Correction
- **Rubberband**: Automatic correction using rubberband algorithm
- **Polynomial**: Correction using configurable order polynomials (1-6)
- **Whittaker**: Penalized least squares smoothing with lambda parameter
- **ALS**: Asymmetric Least Squares with lambda and p parameters
- **ArPLS**: Adaptive reweighted penalized least squares
- **DrPLS**: Doubly reweighted penalized least squares
- **GCV Spline**: Generalized cross-validation spline smoothing
- **Gaussian Process**: Baseline correction using Gaussian processes

### 📊 Normalization
- **Min-Max**: Individual Min-Max spectrum normalization
- **Vector**: L1, L2, or maximum normalization
- **Amida I**: Normalization based on amide I band peak (1600-1700 cm⁻¹)
- **Area**: Area under curve normalization

### 🎯 Smoothing
- **Savitzky-Golay**: Polynomial filter with configurable parameters
- **Wavelets**: Denoising using Daubechies wavelets (db2, db3, db4)
- **Local Polynomial**: LOWESS smoothing with configurable bandwidth
- **Whittaker**: Penalized least squares smoothing
- **GCV Spline**: Generalized cross-validation spline smoothing
- **Flat**: Flat window convolution smoothing
- **Hanning**: Hanning window convolution smoothing

### 📈 Derivatives
- **First Derivative**: First derivative calculation via Savitzky-Golay (order 1)
- **Second Derivative**: Second derivative calculation via Savitzky-Golay (order 2)

### ✂️ Wavelength Truncation
- **Fingerprint Region**: Keep only fingerprint region (900-1800 cm⁻¹)
- **Fingerprint + Lipids**: Keep fingerprint and lipids regions (900-1800, 2800-3050 cm⁻¹)

### 🔍 Model Explainability
- **SHAP Analysis**: Feature importance analysis using SHAP values

## 🏗️ Architecture

```
ftir_prep/
├── core/                    # Core functionalities
│   ├── pipeline.py         # Preprocessing pipeline
│   ├── evaluator.py        # Pipeline evaluation
│   └── explainer.py        # SHAP explainability analysis
├── preprocessing/           # Preprocessing techniques
│   ├── baseline.py         # Baseline correction
│   ├── normalization.py    # Normalization
│   ├── smoothing.py        # Smoothing
│   ├── derivatives.py      # Derivative calculation
│   └── truncation.py      # Wavelength truncation
├── optimization/            # Automatic optimization
│   ├── optuna_optimizer.py # Optuna integration (single-classifier search engine)
│   └── robust_search.py    # RobustPipelineSearch (recommended entry point)
├── utils/                   # Utilities
│   └── data_loader.py      # Data loading
└── config/                  # Configurations
    └── settings.py         # Default parameters
```

## 🚀 Installation

### Requirements
- Python 3.8+
- pip (usually included with Python)

### Installation via PyPI (Recommended - Simplest)


```bash
pip install ftir-prep
```



## 📖 Basic Usage

### 1. Data Loading 
#### 1.1 Separates into groups to guarantee that data from the same patient will be in the same fold in a future classification task
```python
from ftir_prep import FTIRDataLoader

# Load FTIR data
data_loader = FTIRDataLoader(
    data_path="ftir_data.dat",
    wavenumbers_path="wavenumbers.dat"
)

X, y, wavenumbers = data_loader.load_data()

# Create groups var that will be used in classification task to indicate that patient's data must be in the same fold
groups = data_loader.create_groups(instances_per_group=3)
```

#### 1.2 Slices the data to use only one spectra per patient. Data must be ordered by patient
```python
from ftir_prep import FTIRDataLoader

# Load FTIR data
data_loader = FTIRDataLoader(
    data_path="ftir_data.dat",
    wavenumbers_path="wavenumbers.dat"
)

X, y, wavenumbers = data_loader.load_data(slice_size = 3) #use one of the triplicated spectra per patient
```

### 2. Pipeline Creation
```python
from ftir_prep import FTIRPipeline, PipelineBuilder

# Using direct configuration
pipeline = FTIRPipeline()
pipeline.add_step('truncation', 'fingerprint_lipids')
pipeline.add_step('baseline', 'polynomial', polynomial_order=2)
pipeline.add_step('normalization', 'vector')


# Using PipelineBuilder (Fluent API)
pipeline = (PipelineBuilder()
            .add_truncation('fingerprint_lipids')
            .add_baseline('rubberband')
            .add_normalization('minmax')
            .add_smoothing('savgol', polyorder=2)
            .add_derivative('savgol',order=1)
            .build())
```

### 3. Execution and Evaluation
```python
from ftir_prep import PipelineEvaluator

# Process data
X_processed, wavenumbers_processed = pipeline.process(X, wavenumbers)

# Evaluate pipeline
evaluator = PipelineEvaluator(classifier=None, # use default Random Forest
                              cv_method='StratifiedGroupKFold', # cross-validation strategy
                              cv_params={'n_splits': 3, #folds
                                        'shuffler': False,
                                        'random_state': 42})
results = evaluator.evaluate_pipeline(pipeline,
                                      X, y,
                                      groups, # groups var created previously
                                      wavenumbers=wavenumbers
)

print(f"Accuracy: {results['mean_accuracy']:.4f} ± {results['std_accuracy']:.4f}")
```

### 4. Automatic Optimization

`RobustPipelineSearch` is the recommended way to find an optimal preprocessing
pipeline: it runs an Optuna search per classifier in a pool, cross-validates
the resulting candidates on the *other* classifiers in that pool, and picks
the pipeline that is most consistently good across all of them — more likely
to generalize than one tuned for a single model.

Only `X`, `y` and `wavenumbers` are required. Everything else — including the
classifier pool — falls back to sensible defaults:

```python
from ftir_prep import RobustPipelineSearch, load_pipeline

selector = RobustPipelineSearch(X, y, wavenumbers)

best_pipeline, final_report = selector.run(n_trials=1000)  # or timeout=<seconds>

print(best_pipeline.steps)
print(final_report)  # validation score per classifier
```

Pass `groups=groups` with `cv_method='StratifiedGroupKFold'` (see Data Loading
above) for group-based cross-validation instead of the `StratifiedKFold`
default.

By default `RobustPipelineSearch` searches against a predefined pool of ~6-7
classifiers (LogisticRegression, SVC, ExtraTreesClassifier,
KNeighborsClassifier, MLPClassifier, GaussianNB, and XGBClassifier if
`xgboost` is installed) and picks the pipeline that generalizes best across
all of them. Pass `classifiers` to use a **single model** instead (no
cross-classifier robustness step then — just the best pipeline found for
that model) or **your own custom pool**:

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

# Single model
selector = RobustPipelineSearch(
    X, y, wavenumbers,
    classifiers={"RandomForest": RandomForestClassifier(random_state=42)},
)

# Custom pool (with exactly 2 classifiers, candidates are ranked by the
# plain average of both scores instead of the usual robustness formula)
selector = RobustPipelineSearch(
    X, y, wavenumbers,
    classifiers={
        "RandomForest": RandomForestClassifier(random_state=42),
        "LogisticRegression": LogisticRegression(random_state=42, max_iter=1000),
        "SVC": SVC(),
    },
)
```

It also exposes **time, trial and memory budgets**, parallelization, and
checkpointing to disk — useful for long, unattended runs:

```python
selector = RobustPipelineSearch(
    X, y, wavenumbers,
    core_budget=4,       # cores for parallel search (default: sequential)
)

best_pipeline, final_report = selector.run(
    timeout=1800,             # wall-clock seconds, instead of n_trials
    memory_budget_mb=8192,    # optional RAM ceiling
    output_dir="search_results",  # checkpoints every stage to disk as it runs
)

# Reload the winning pipeline later, in another session
best_pipeline = load_pipeline("search_results/best_pipeline/best_pipeline.json")
```

`n_trials` and `timeout` are mutually exclusive. For finer control, each
stage (`search()`, `select_candidates()`, `cross_validate_candidates()`,
`rank_candidates()`, `select_best()`) can also be called individually instead
of `run()` — see `examples/pipeline_search/pipeline_search.py`.

### 5. Model Explainability
```python
from ftir_prep import FTIRExplainer

# Create explainer
explainer = FTIRExplainer(classifier=your_classifier)

# Analyze feature importance with SHAP
# It will save in output_dir a csv and a png with feature importance data
results = explainer.explain_model(
    X_processed, y, groups,
    split_method='stratified_group',
    feature_names=wavenumbers_processed,
    output_dir="shap_analysis"
)
```

## 🔬 Practical Examples

### Pipeline Creation Examples
```bash
# Direct configuration example
python3 examples/create_pipeline/direct_configuration.py

# PipelineBuilder (Fluent API) example
python3 examples/create_pipeline/pipeline_builder.py
```

### Pipeline Comparison Example
```bash
# Compare different preprocessing strategies
python3 examples/compare_pipelines/compare_pipelines.py
```

### Pipeline Optimization Example
```bash
# Automatic pipeline optimization
python3 examples/pipeline_search/pipeline_search.py
```

### Pipeline Loading Example
```bash
# Load and use saved pipelines
python3 examples/read_pipeline_from_file/read_pipeline_file.py
```

### SHAP Explainability Example
```bash
# Feature importance analysis with SHAP
python3 examples/shap_analysis/explainer_example.py
```

## 🎯 Use Cases

### Disease Diagnosis
- Analysis of FTIR spectra from biological samples
- Biomarker identification
- Automatic sample classification

### Scientific Research
- Methodology comparison
- Protocol optimization
- Result validation


## 📚 Documentation

- **Docstrings**: Complete inline documentation
- **Examples**: Functional example code

## 👥 Authors

- **Lucas Mendonça** - *Initial development* - [GitHub](https://github.com/lucas-mendonca-andrade)


⭐ If this project was useful to you, consider giving it a star on GitHub! 
