Metadata-Version: 2.4
Name: nmrmetaproc
Version: 1.0.35
Summary: NMR Metabolomics Spectral Processor - raw Bruker FID to analysis-ready CSV
Author-email: Folorunsho Bright Omage <omagefolorunsho@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/omagebright/nmrmetaproc
Project-URL: Repository, https://github.com/omagebright/nmrmetaproc
Keywords: NMR,metabolomics,spectral processing,chemometrics,Bruker
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Chemistry
Requires-Python: <3.14,>=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
License-File: AUTHORS.md
Requires-Dist: nmrglue>=0.9
Requires-Dist: numpy<2.3,>=1.21
Requires-Dist: scipy>=1.7
Requires-Dist: pandas>=1.3
Requires-Dist: tqdm>=4.60
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: PyQt6>=6.5; extra == "dev"
Requires-Dist: pyqtgraph>=0.13; extra == "dev"
Requires-Dist: PyYAML>=6.0; extra == "dev"
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
Provides-Extra: gui
Requires-Dist: PyQt6>=6.5; extra == "gui"
Requires-Dist: pyqtgraph>=0.13; extra == "gui"
Requires-Dist: PyYAML>=6.0; extra == "gui"
Provides-Extra: benchmark
Requires-Dist: openpyxl>=3.1; extra == "benchmark"
Requires-Dist: rdata>=1.0; extra == "benchmark"
Dynamic: license-file

# nmrmetaproc

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python](https://img.shields.io/badge/python-3.9--3.13-blue)](https://www.python.org/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19194107.svg)](https://doi.org/10.5281/zenodo.19194107)

**NMR Metabolomics Spectral Processor**

`nmrmetaproc` converts raw Bruker NMR FID files into clean, analysis-ready spectral matrices (CSV format) suitable for PCA, PLS-DA, pathway analysis, and other downstream metabolomics workflows. It implements a rigorous, reproducible processing pipeline with automatic phase correction, chemical-shift referencing, robust baseline correction, spectral alignment, and Probabilistic Quotient Normalization (PQN).

**Authors:** Folorunsho Bright Omage, Toyin Bright Omage, Ljubica Tasic  
**ORCID:** [0000-0002-9750-5034](https://orcid.org/0000-0002-9750-5034)  
**Email:** omagefolorunsho@gmail.com  
**Provenance:** [Zenodo DOI](https://doi.org/10.5281/zenodo.19194107), [CITATION.cff](CITATION.cff), [AUTHORS.md](AUTHORS.md), [NOTICE](NOTICE), [PROVENANCE.md](PROVENANCE.md)

---

## Features

- Reads raw Bruker FID files (`fid`/`*.fid` + `acqus`/`acqu`) directly, no conversion needed
- Automatically discovers valid Bruker spectra under the opened directory (that folder is the study) and splits mixed data into scientifically comparable batches by pulse program
- Applies pulse-program-specific and sample-context-aware processing presets for common Bruker 1D 1H experiments (`noesygppr1d` including `noesy1d`/`noesyd` aliases, `cpmgpr1d`, `zg30`, `ledbpgppr2s1d`)
- Full processing pipeline in correct order:
  1. Exponential apodization (line broadening)
  2. Zero-filling
  3. Fast Fourier Transform
  4. **Automatic phase correction** (ACME algorithm, no fixed phase values)
  5. Context-aware chemical-shift referencing to TSP/DSS, TMS, residual CHCl3, or lipid methylene as appropriate
  6. Asymmetric least-squares (ALS) baseline correction
  7. Negative-value handling with per-sample logging
  8. Solvent/context region exclusion, for example aqueous water or residual CHCl3
  9. Optional spectral alignment (one whole-spectrum correlation shift against
     the pointwise cohort median, or reference-peak alignment)
  10. Configurable region exclusion
  11. Uniform binning
  12. **PQN normalization** (default), total area, or none. Pipeline-level TSP
      normalization is rejected until validated pre-exclusion reference
      quantification is available.
- Per-sample quality control: SNR, reference-peak linewidth, solvent-region score, clean pass/warn/fail separation, and optional adaptive second-pass audit
- Clean CSV outputs ready for MetaboAnalyst, R, MATLAB, machine learning, and chemometrics
- Works on Windows, macOS, and Linux

---

## Download

Pre-built desktop applications for Windows, macOS, and Linux are published on the [Releases page](https://github.com/omagebright/nmrmetaproc/releases). Research-beta releases are marked clearly there.

| Platform | File | How to use |
|----------|------|-----------|
| Windows 10 / 11 (x64) | `nmrmetaproc-gui-windows-x64.zip` | Unzip and run `nmrmetaproc-gui.exe` |
| macOS 12+ (architecture stated per release) | `nmrmetaproc-gui-macos.dmg` | Open the DMG, drag `nmrmetaproc.app` to Applications |
| Linux (glibc 2.31+, x86_64) | `nmrmetaproc-gui-linux-x86_64.AppImage` | `chmod +x` the file and double-click |

These bundles contain Python, all dependencies, and the GUI app. You do **not** need to install Python separately to use them.

If you prefer Python integration (CLI + scriptable API), install from PyPI instead -- see [Installation](#installation) below.

### First-launch notes

The desktop bundles are **unsigned and not notarized** -- we do not yet hold an Apple Developer ID or a Windows EV code-signing certificate, so the operating system shows a warning the first time you open the app. They are built from the release tag with bundled dependencies. Before installing, verify the download against the `SHA256SUMS.txt` file attached to that release.

**macOS (Gatekeeper)** -- if you see *"Apple could not verify nmrmetaproc is free of malware"* with no Open button, do **one** of the following after dragging the .app to /Applications:

- **Terminal one-liner** (fastest):
  ```bash
  xattr -dr com.apple.quarantine /Applications/nmrmetaproc.app
  ```
  Then launch normally.

- **GUI path**: click **Done** on the dialog (do **not** click Move to Bin), then open **System Settings -> Privacy & Security**, scroll to the **Security** section, click **Open Anyway** next to the "nmrmetaproc was blocked" line, and re-launch the app.

**Windows (SmartScreen)** -- on first launch you may see *"Windows protected your PC"*. Click **More info**, then **Run anyway**.

**Linux (AppImage permissions)** -- the file needs the executable bit set:
```bash
chmod +x nmrmetaproc-gui-linux-x86_64.AppImage
```
If your distro complains about FUSE, install `libfuse2` (Ubuntu: `sudo apt install libfuse2`).

---
## Installation

```bash
pip install nmrmetaproc
```

Or from source:

```bash
git clone https://github.com/omagebright/nmrmetaproc.git
cd nmrmetaproc
pip install -e .
```

**Dependencies:** `nmrglue`, `numpy`, `scipy`, `pandas`, `tqdm`

Supported Python versions are 3.9 through 3.13. Python 3.14 is excluded because
the verified scientific dependency stack does not yet produce reliable numeric
results there.

---

---

## Try It Now - Public Benchmark Demo

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/omagebright/f989c18c1e7fc74461af536f49602ae9/colab_demo_real_data.ipynb)

**Try nmrmetaproc instantly** with public, attributable NMR data. The interactive Google Colab notebook installs `nmrmetaproc==1.0.35` from PyPI (the same version as the CLI and desktop apps). The repository is private, so the badge opens a public gist copy that anonymous users can load. It does not fetch a private GitHub zip. After PyPI has 1.0.35, `pip install nmrmetaproc==1.0.35` is the only install step.

**Demo features:**
- Download public Bruker HumanSerum FIDs from PepsNMRData
- Check the published `FinalSpectra_HS` matrix shape
- Run NMRMetaProc and write an auditable benchmark JSON
- Show QC diagnostics without inventing source pass/fail labels
- Zero installation required - runs in your browser

[**Launch Demo**](https://colab.research.google.com/gist/omagebright/f989c18c1e7fc74461af536f49602ae9/colab_demo_real_data.ipynb)

## Command-Line Usage

### Full Processing Pipeline

Point `nmrmetaproc` at the study directory. By default it discovers all valid spectra, treats that folder as one cohort, separates incompatible pulse programs into independent batches, applies the registered pulse-program presets, and writes one analysis-ready matrix per batch.

```bash
nmrmetaproc process /path/to/bruker/data --output ./results
```

To run the transparent adaptive second pass from the CLI, add:

```bash
nmrmetaproc process /path/to/bruker/data --output ./results --adaptive-second-pass
```

The first-pass QC remains available as `qc_report_initial.csv`; the selected
second-pass candidate and before/after QC metrics are recorded in
`adaptive_qc_report.csv`. A corrected pass requires an improved rerun QC with
a measurable reference peak; candidates that lose reference evidence are rejected.

Batch output layout:

```text
results/
|-- batch_manifest.csv
|-- public_serum__zg30/
|   |-- spectral_matrix.csv
|   |-- qc_report.csv
|   |-- qc_report_initial.csv              # when adaptive second pass is enabled
|   |-- adaptive_qc_report.csv             # when adaptive second pass is enabled
|   |-- acquisition_parameters.csv
|   `-- processing.log
`-- public_urine__noesygppr1d/
    |-- spectral_matrix.csv
    |-- qc_report.csv
    |-- qc_report_initial.csv              # when adaptive second pass is enabled
    |-- adaptive_qc_report.csv             # when adaptive second pass is enabled
    |-- acquisition_parameters.csv
    `-- processing.log
```

Use `--single-matrix` only when the input directory is already one homogeneous cohort and pulse program:

```bash
nmrmetaproc process /path/to/one_batch --output ./results --single-matrix
```

### QC Scan Only

```bash
nmrmetaproc qc /path/to/data --output ./qc_results
```

### Inspect Available Samples

```bash
nmrmetaproc info /path/to/data
```

---

## Python API

```python
from nmrmetaproc import NMRProcessor

processor = NMRProcessor(
    lb=0.3,
    bin_width=0.01,
    normalization="pqn",
    ppm_range=(0.5, 9.5),
    snr_threshold=10.0,
    linewidth_threshold=2.5,
    align="correlation",  # optional; robust cohort-median target
)

results = processor.process("/path/to/bruker/data")

print(results.spectral_matrix)   # rows=samples, columns=ppm bins
print(results.qc_report)         # SNR, linewidth, pass/fail per sample

results.save("./output")
```

---

## Output Files

| File | Description |
|------|-------------|
| `spectral_matrix.csv` | Rows = samples (passed QC), columns = ppm bin centres. PQN/total-area normalized; **not** scaled |
| `spectral_matrix_labeled.csv` | Optional. Labelled feature matrix (only when peak-label export is enabled). Auto selects LipSpin (CDCl3 lipid extract) or the curated aqueous HMDB overlay from solvent/matrix, not from pulse program. Solvent/reference labels and overlapping 1D assignment windows are excluded |
| `spectral_matrix_analysis_ready.csv` | Optional. Low-information bins removed, robust per-bin outlier screen, **robust feature scaling already applied** (median-centred, MAD-scaled), so do **not** autoscale again in MetaboAnalyst/sklearn. Robust steps are skipped for cohorts with < 8 samples |
| `analysis_ready_report.csv` | Optional. Audit table for the raw-bin analysis-ready matrix |
| `spectral_matrix_labeled_analysis_ready.csv` | Optional. ML/chemometrics-ready version of the labelled matrix, written when both label export and analysis-ready export are enabled |
| `analysis_ready_labeled_report.csv` | Optional. Audit table for the labelled analysis-ready matrix |
| `qc_report.csv` | Final QC used for matrix inclusion. Corrected samples are marked with adaptive stage/status columns when adaptive QC is enabled |
| `qc_report_initial.csv` | Optional. Untouched first-pass QC before adaptive second-pass reprocessing |
| `adaptive_qc_report.csv` | Optional. Before/after second-pass audit table: trigger, candidate, score delta, SNR, linewidth, residual score, reference ppm/QC, warnings |
| `acquisition_parameters.csv` | Bruker metadata, inferred context/solvent, pulse program, effective per-sample processing settings, and adaptive candidate settings when applied |
| `processing_log.txt` | Full processing log with all parameters and per-sample status (batch mode writes the same log as `processing.log` inside each cohort directory) |

---

## Data Format

Each sample must be in its own directory containing:
- `fid` or `*.fid` - binary FID data (interleaved real/imaginary int32)
- `acqus` or `acqu` - acquisition parameter file

```
data_root/
|-- sample_001/
|   |-- fid
|   `-- acqus
`-- sample_002/
    |-- fid
    `-- acqus
```

Nested layouts are also supported and discovered automatically.

---

## Alignment contract

`align="correlation"` applies one integer, whole-spectrum shift with edge
padding. The pipeline targets the pointwise cohort median and searches the
0.02 ppm window resolved against the actual axis. The median limits a minority
composition outlier's influence on its target; it does not
turn whole-spectrum alignment into region-dependent warping or prove that every
chosen shift is chemically correct. Use `align=None` (CLI: `--align none`) when
a cohort should retain its referenced axes without mutual alignment.

The processing log records the reference strategy and applied window. A
`WINDOW SATURATED` line means only that an unconstrained correlation maximum lay
outside that window. Its absence is not a correctness signal: the harmful
ST000049 cohort-mean shift was inside 0.02 ppm and produced no warning. A known
small rigid-drift positive control and the public ST000049 regression receipt
are both tested; larger, fractional, or region-dependent drift is not promised.
The lower-level `correlation_shift_align(reference=None)` function retains its
cohort-mean target for compatibility, while callers can pass an explicit target.

## Reference and QC notes

The software chooses the reference from both pulse program and sample context. Aqueous biofluids use TSP/DSS at 0.000 ppm. Non-aqueous spectra with explicit TMS evidence use TMS at 0.000 ppm. CDCl3 lipid extracts without TMS evidence use residual CHCl3 at 7.260 ppm and exclude 7.24-7.28 ppm from the matrix. If `pdata/1/procs` or `pdata/1/proc` is present, NMRMetaProc uses Bruker `OFFSET`, `SW_p`, and `SF` to construct the initial ppm-axis geometry and records `ppm_axis_source=bruker_procs` in `acquisition_parameters.csv`; metadata presence alone is not reported as a reference lock. With referencing enabled, the selected peak is measured on either axis source. A robust measured peak can correct a processed axis, while an unverified processed axis is retained unchanged and explicitly reported as `reference_locked=false` with a `REFERENCE UNVERIFIED` warning. A failed acquisition-axis lock is likewise reported, with acceptance depending on the selected reference and context policy rather than an assumption of calibration. QC SNR is computed from the main metabolite window after excluding the active residual solvent/window, so a bad water peak cannot inflate the signal score. The residual-region score is the residual maximum divided by the maximum non-residual signal; values above 1.0 mean the residual is stronger than any non-residual signal. Moderate residual water in an excluded aqueous water window remains an exported diagnostic, while severe residual dominance becomes a warning. WARN samples are accepted into the matrix but carry review items; clean PASS, WARN, and FAIL are reported separately in the GUI and QC report.

Public benchmark rules are recorded in [benchmarks/PUBLIC_BENCHMARKS.md](benchmarks/PUBLIC_BENCHMARKS.md) and [benchmarks/public_benchmark_manifest.json](benchmarks/public_benchmark_manifest.json). Private or collaborator samples are not default benchmarks. A sample can be called source-published pass, treatable, or fail only when the exact label and source URL are recorded in the manifest.

A public Bruker 1D FID fixture from the MMCD/nmrglue example mirror is included in the test suite to validate raw FID reading, PULPROG alias dispatch, ppm-axis direction, and dominant solvent-region placement.

For a biological metabolomics benchmark, `benchmarks/pepsnmr_human_serum/` provides a reproducible runner for the public Bioconductor PepsNMRData package. It downloads raw Bruker human-serum FIDs plus the package's published `FinalSpectra_HS` matrix, processes the raw FIDs with NMRMetaProc, and writes an auditable benchmark JSON. PepsNMRData does not provide human QC pass/fail labels, so the benchmark checks external discovery, pulse-program dispatch, finite matrix export, accepted-row agreement with the published 32-sample processed matrix, and transparent QC diagnostics rather than forcing artificial QC labels.

The 2026-07-26 scientific run identified and fixed a Bruker convention defect:
`GRPDLY=-1` means derive the digital-filter delay from `DECIM`/`DSPFVS`, not
skip correction. Against `Re(FinalSpectra_HS)`, the corrected 32-sample output
reached median Pearson 0.9800 (minimum 0.9587), median cosine 0.9820 (minimum
0.9624), median absolute residual shift 0.0025 ppm, and 95th-percentile absolute
residual shift 0.0040 ppm. These pass the criteria declared before the fix in
the independent benchmark oracle.

All 32 linewidth estimates remain unreliable and are surfaced as
WARN/indeterminate rather than silent clean passes. Because PepsNMRData
publishes no sample-level pass/treatable/fail labels, this validates the
declared spectral-agreement scope and transparent status handling only; it does
not establish biological QC accuracy, linewidth calibration, or production
readiness.

The independent runner in `benchmarks/metabolomics_workbench/` extends the
preprocessing evidence to two checksum-bound, public-domain-after-embargo
datasets without relaxing the PepsNMRData criteria. The recorded v1.0.30
ST000049 run processed and exactly mapped 13/13 insect-extract, 950 MHz
`noesypr1d` FIDs, reaching median Pearson 0.9568 and median cosine 0.9636 against
248 usable depositor bins. A later 0.02 ppm cohort-mean alignment default
regressed `Control_X` to Pearson/cosine 0.819528/0.862030 without a saturation
warning. The current pointwise-median pipeline reference passes the complete
checksum-bound rerun at the same window: minimum Pearson/cosine
0.936941/0.945661, `Control_X` 0.956578/0.963597, and median/p95 residual shift
0.000/0.000 ppm. The reduced public receipt and full rerun instructions are in
`tests/data/st000049_alignment_receipt/`.

The recorded v1.0.30 ST000306 run processed 113/113 human-plasma, 700 MHz instrument-declared
`cpmgpr1d` FIDs and exactly mapped all 110 depositor rows, reaching median
Pearson 0.9690, median cosine 0.9735, and median/p95 absolute shift
0.006/0.016 ppm against 142 usable bins.

Both studies identify repeated pooled material, and both the published and
regenerated matrices preserve higher within-pool similarity than ordinary
sample-pair similarity. This is evidence for technical repeatability only.
Neither source publishes per-sample pass/treatable/fail labels, so these runs do
not validate biological QC classification, universal thresholds, or production
readiness. See [the Workbench benchmark documentation](benchmarks/metabolomics_workbench/README.md)
for checksums, exclusions, commands, and source limitations.

To rerun the local automated public-evidence gate after the benchmark output exists:

```bash
python benchmarks/validate_public_release.py
```

The checksum-bound ST000020 urine benchmark independently re-integrates every
deposited `pdata/1/1r` spectrum before comparing NMRMetaProc with the depositor
matrix. The guard identifies one predeclared source-invalid row and then
compares the remaining 107 rows symmetrically. Under the documented
source-compatible urine workflow, it reaches median/minimum Pearson
0.99418/0.96385 and median/minimum cosine 0.99502/0.96562. Its pooled samples
remain a repeatability proxy only: the source does not publish sample-level
pass/treatable/fail labels.

---

## Citing and Provenance

If you use `nmrmetaproc` in your research, please cite:

```
Omage, F. B., Omage, T. B., & Tasic, L. (2026). nmrmetaproc: NMR Metabolomics Spectral Processor (Version 1.0.35).
Zenodo. https://doi.org/10.5281/zenodo.19194107
```

The canonical public provenance records are the GitHub repository, annotated
release tags, GitHub Releases, [CITATION.cff](CITATION.cff), [AUTHORS.md](AUTHORS.md),
[NOTICE](NOTICE), [PROVENANCE.md](PROVENANCE.md), and the Zenodo DOI. The MIT
License permits reuse and redistribution only with the required copyright and
permission notices retained. Scientific use should preserve authorship and cite
the software DOI.

The PQN normalization method:

> Dieterle, F., Ross, A., Schlotterbeck, G., & Senn, H. (2006). Probabilistic quotient
> normalization as robust method to account for dilution of complex biological mixtures.
> *Analytical Chemistry*, 78(13), 4281-4290. https://doi.org/10.1021/ac051632c

---

## Development

```bash
git clone https://github.com/omagebright/nmrmetaproc.git
cd nmrmetaproc
pip install -e ".[dev]"
pytest tests/ -v
```

---

## License

MIT License. See [LICENSE](LICENSE) for details. Redistribution and derivative
work must retain the copyright and permission notices. See [NOTICE](NOTICE) and
[PROVENANCE.md](PROVENANCE.md) for authorship and citation records.
