Metadata-Version: 2.4
Name: mutcleaner
Version: 0.2.0
Summary: An efficient framework for cleaning, standardizing, and processing biological mutation data.
Author: Yuxiang Tang, Ziyu Shi
License-Expression: BSD-3-Clause
Project-URL: Repository, https://github.com/xulab-research/MutCleaner
Keywords: mutation,sequence,bioinformatics,data-cleaning,standardization,protein,codon,DNA,RNA,DMS
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: joblib>=1.5.0
Requires-Dist: numpy>=2.1.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: tqdm>=4.60.0
Requires-Dist: requests>=2.30
Requires-Dist: openpyxl>=3.1.0
Provides-Extra: test
Requires-Dist: pytest>=8.0.0; extra == "test"
Requires-Dist: pytest-cov>=6.0.0; extra == "test"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=6.0.0; extra == "dev"
Requires-Dist: sphinx>=8.0.0; extra == "dev"
Requires-Dist: sphinx-autobuild>=2024.10.0; extra == "dev"
Requires-Dist: pydata-sphinx-theme; extra == "dev"
Requires-Dist: twine>=6.0.0; extra == "dev"
Requires-Dist: gitpython; extra == "dev"
Requires-Dist: pygithub; extra == "dev"
Requires-Dist: jinja2; extra == "dev"
Requires-Dist: numpydoc; extra == "dev"
Requires-Dist: myst_parser; extra == "dev"
Requires-Dist: sphinx_copybutton; extra == "dev"
Requires-Dist: sphinx_design; extra == "dev"
Requires-Dist: sphinx_autodoc_typehints; extra == "dev"
Requires-Dist: sphinxcontrib-mermaid; extra == "dev"
Dynamic: license-file

# MutCleaner

[![License badge](https://img.shields.io/badge/License-BSD_3--Clause-yellow?logo=opensourceinitiative&logoColor=white)](https://github.com/xulab-research/MutCleaner/blob/main/LICENSE)
[![PyPI version badge](https://img.shields.io/pypi/v/mutcleaner.svg?logo=python&logoColor=white&color=orange)](https://pypi.org/project/mutcleaner/)
[![Docs](https://github.com/xulab-research/MutCleaner/actions/workflows/docs.yml/badge.svg)](https://xulab-research.github.io/MutCleaner/)

MutCleaner is an extensible Python toolkit for cleaning and standardizing biological mutation datasets, integrating dataset-specific cleaning pipelines with core abstractions for protein, nucleotide, and codon-level mutation representations.

* **Documentation**: https://xulab-research.github.io/MutCleaner
* **Cleaning Examples**: https://xulab-research.github.io/MutCleaner/user_guide/supported_datasets.html

## Overview

MutCleaner is an extensible Python toolkit for cleaning, standardizing, and analyzing biological mutation datasets. It currently focuses on protein variant data while providing core abstractions for DNA, RNA, protein sequences, and codon-level mutation representations.

The package combines dataset-specific cleaning pipelines with reusable sequence and mutation utilities, enabling reproducible preprocessing of large-scale mutational datasets for downstream bioinformatics and machine learning analyses.

### Key Capabilities

- **Mutation dataset cleaning and standardization**: Harmonize mutation annotations, sequences, labels, and metadata across heterogeneous biological mutation datasets.
- **Sequence representation and validation**: Utilities for DNA, RNA, and protein sequences, including validation, transcription, reverse transcription, translation, and mutation application.
- **Mutation parsing and transformation**: Tools for parsing amino-acid and codon-level mutations, inferring mutations from sequences, applying mutations to reference sequences, and converting codon mutations into amino-acid changes.
- **Modular pipeline architecture**: A composable pipeline interface for building reproducible dataset-cleaning workflows.
- **Parallel and scalable dataset processing**: Multi-core utilities for mutation validation, mutation application, and sequence-based mutation inference, supporting efficient processing of large tabular mutation datasets.

## Installation

### Requirements

- Python 3.13+
- Dependencies are automatically installed via pip.

### Install via pip

```bash
pip install mutcleaner
```

### Development Installation

```bash
git clone https://github.com/xulab-research/MutCleaner.git MutCleaner
cd MutCleaner
pip install -e .
```

To install development dependencies for testing and documentation:

```bash
pip install -e ".[dev]"
```

## Package Structure
```text
mutcleaner/
├── cleaners/                # Reusable cleaners and dataset-specific pipelines
│   ├── basic_cleaners.py    # Reusable basic cleaning functions
│   ├── base_config.py       # Shared cleaner configuration
│   └── *_cleaner.py         # Dataset-specific cleaning pipelines
├── core/                    # Core data structures and processing logic
│   ├── alphabet.py          # Biological alphabet definitions and validation
│   ├── codon.py             # Codon table and sequence translation utilities
│   ├── constants.py         # Shared biological constants
│   ├── dataset.py           # Standard dataset abstraction and export logic
│   ├── mutation.py          # Mutation parsing, representation and validation
│   ├── pipeline.py          # Reusable data cleaning pipeline framework
│   ├── sequence.py          # DNA, RNA and protein sequence abstractions
│   └── types.py             # Shared type aliases and annotations
└── utils/                   # General helper functions for downloading and I/O
```
### Module Overview
#### mutcleaner.cleaners

The `cleaners` module provides both reusable cleaning functions and predefined dataset-specific cleaning pipelines.  

`basic_cleaners.py` contains general-purpose cleaning functions that can be reused across different datasets, such as column checking, missing value handling, sequence validation, mutation validation, and common formatting operations.  
`base_config.py` defines shared configuration used by dataset cleaners, such as common column names, required fields, and reusable cleaner settings.  
`*_cleaner.py` files define dataset-specific cleaning pipelines. Each file is designed for a particular dataset or benchmark and combines reusable cleaning functions into a complete workflow.  
#### mutcleaner.core

The `core` module contains the fundamental data structures and processing logic used throughout MutCleaner.  

`alphabet.py` defines biological alphabets and validation rules for DNA, RNA, and protein sequences.  
`codon.py` provides codon table utilities and sequence translation functionality.
`constants.py` stores shared biological constants, such as amino acid symbols, nucleotide symbols, complements, and stop codon definitions.  
`dataset.py` defines the standard dataset abstraction used to store, manage, validate, and export cleaned mutation datasets.  
`mutation.py` provides mutation parsing, mutation representation, and mutation validation logic.  
`pipeline.py` defines the reusable data cleaning pipeline framework, including pipeline steps, execution order, and artifact tracking.  
`sequence.py` defines biological sequence abstractions, including DNA, RNA, and protein sequence classes.  
`types.py` stores shared type aliases and annotations used across the package.  

#### mutcleaner.utils

The `utils` module contains helper functions that support common operations outside the core cleaning logic.  
It includes utilities for downloading source files, handling paths, extracting files, checking file existence, and managing common input/output operations.  
## Quick Start

See the [Data Cleaners Usage Guide](https://xulab-research.github.io/MutCleaner/user_guide) for more examples.

### Supported Datasets

<table>
  <thead>
    <tr>
      <th>Dataset Name</th>
      <th>Reference</th>
      <th>File</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Human Domainome Dataset</td>
      <td><a href="https://doi.org/10.1038/s41586-024-08370-4">Site-saturation mutagenesis of 500 human protein domains</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/Human_Domainome_Dataset/SupplementaryTable2.txt">SupplementaryTable2.txt</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/Human_Domainome_Dataset/wild_type.fasta">wild_type.fasta</a></td>
    </tr>
    <tr>
      <td>ProteinGym DMS Substitutions Dataset</td>
      <td><a href="https://doi.org/10.1101/2023.12.07.570727">ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/ProteinGym_DMS_Substitutions_Dataset/DMS_ProteinGym_substitutions.zip">DMS_ProteinGym_substitutions.zip</a></td>
    </tr>
    <tr>
      <td>cDNA Proteolysis Dataset</td>
      <td><a href="https://doi.org/10.1038/s41586-023-06328-6">Mega-scale experimental analysis of protein folding stability in biology and design</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/cDNA_Proteolysis_Dataset/Tsuboyama2023_Dataset2_Dataset3_20230416.csv">Tsuboyama2023_Dataset2_Dataset3_20230416.csv</a></td>
    </tr>
    <tr>
      <td>ddG Dataset</td>
      <td><a href="https://doi.org/10.1038/s43588-024-00716-2">Improving the prediction of protein stability changes upon mutations by geometric learning and a pre-training strategy</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94%CE%94G_Dataset/M1261.csv">M1261.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94%CE%94G_Dataset/S461.csv">S461.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94%CE%94G_Dataset/S669.csv">S669.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94%CE%94G_Dataset/S783.csv">S783.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94%CE%94G_Dataset/S8754.csv">S8754.csv</a></td>
    </tr>
    <tr>
      <td>dTm Dataset</td>
      <td><a href="https://doi.org/10.1038/s43588-024-00716-2">Improving the prediction of protein stability changes upon mutations by geometric learning and a pre-training strategy</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94Tm_Dataset/S4346.csv">S4346.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/%CE%94Tm_Dataset/S571.csv">S571.csv</a></td>
    </tr>
    <tr>
      <td>ArchStabMS1E10 Epistasis Dataset</td>
      <td><a href="https://doi.org/10.1038/s41586-024-07966-0">The genetic architecture of protein stability</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/ArchStabMS1E10_Epistasis_Dataset/ArchStabMS1E10_Epistasis_Dataset.csv">ArchStabMS1E10_Epistasis_Dataset.csv</a></td>
    </tr>
    <tr>
      <td>Antitoxin ParD3 Epistasis Dataset</td>
      <td><a href="https://doi.org/10.1038/s41467-024-45621-4">Protein design using structure-based residue preferences</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/Antitoxin_ParD3_Epistasis_Dataset/Antitoxin_ParD3_Epistasis_Dataset.csv">Antitoxin_ParD3_Epistasis_Dataset.csv</a></td>
    </tr>
    <tr>
      <td>TrpB Epistasis Dataset</td>
      <td><a href="https://doi.org/10.1073/pnas.2400439121">A combinatorially complete epistatic fitness landscape in an enzyme active site</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/TrpB_Epistasis_Dataset/TrpB_Epistasis_Dataset.csv">TrpB_Epistasis_Dataset.csv</a></td>
    </tr>
    <tr>
      <td>Human Myoglobin Epistasis Dataset</td>
      <td><a href="https://doi.org/10.1101/2024.02.24.581358">Decoding Stability and Epistasis in Human Myoglobin by Deep Mutational Scanning and Codon-level Machine Learning</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/Human_Myoglobin_Epistasis_Dataset/Human_Myoglobin_Epistasis_Dataset.csv">Human_Myoglobin_Epistasis_Dataset.csv</a></td>
    </tr>
    <tr>
      <td>CTXM Epistasis Dataset</td>
      <td><a href="https://doi.org/10.1073/pnas.2313513121">Network of epistatic interactions in an enzyme active site revealed by DMS</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/CTXM_Epistasis_Dataset/CTXM_Cefotaxime_Epistasis_Dataset.csv">CTXM_Cefotaxime_Epistasis_Dataset.csv</a>, <a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/CTXM_Epistasis_Dataset/CTXM_Ampicillin_Epistasis_Dataset.csv">CTXM_Ampicillin_Epistasis_Dataset.csv</a></td>
    </tr>
    <tr>
      <td rowspan="4" valign="middle">RBD ACE2 Dataset</td>
      <td><a href="https://doi.org/10.1126/science.abo7896">Shifting mutational constraints in the SARS-CoV-2 receptor-binding domain during viral evolution</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_ACE2_Dataset/SARS-CoV-2-RBD_DMS_variants_bc_binding.csv">SARS-CoV-2-RBD_DMS_variants_bc_binding.csv</a><br><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_ACE2_Dataset/SARS-CoV-2-RBD_Delta_bc_binding.csv">SARS-CoV-2-RBD_Delta_bc_binding.csv</a></td>
    </tr>
    <tr>
      <td><a href="https://doi.org/10.1371/journal.ppat.1010951">Deep mutational scans for ACE2 binding, RBD expression, and antibody escape in the SARS-CoV-2 Omicron BA.1 and BA.2 receptor-binding domains</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_ACE2_Dataset/SARS-CoV-2-RBD_DMS_Omicron_bc_binding.csv">SARS-CoV-2-RBD_DMS_Omicron_bc_binding.csv</a></td>
    </tr>
    <tr>
      <td><a href="https://doi.org/10.1371/journal.ppat.1011901">Deep mutational scans of XBB.1.5 and BQ.1.1 reveal ongoing epistatic drift during SARS-CoV-2 evolution</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_ACE2_Dataset/SARS-CoV-2-RBD_DMS_Omicron-XBB-BQ_bc_binding.csv">SARS-CoV-2-RBD_DMS_Omicron-XBB-BQ_bc_binding.csv</a></td>
    </tr>
    <tr>
      <td><a href="https://doi.org/10.1093/ve/veae067">Deep mutational scanning of SARS-CoV-2 Omicron BA.2.86 and epistatic emergence of the KP.3 variant</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_ACE2_Dataset/SARS-CoV-2-RBD_DMS_Omicron-EG5-FLip-BA286_bc_binding.csv">SARS-CoV-2-RBD_DMS_Omicron-EG5-FLip-BA286_bc_binding.csv</a></td>
    </tr>
    <tr>
      <td rowspan="3" valign="middle">RBD Antibody Dataset</td>
      <td><a href="https://doi.org/10.1126/scitranslmed.abi9915">The SARS-CoV-2 mRNA-1273 vaccine elicits more RBD-focused neutralization, but with broader antibody binding within the RBD</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_Antibody_Dataset/SARS-CoV-2-RBD_MAP_Moderna.csv">SARS-CoV-2-RBD_MAP_Moderna.csv</a></td>
    </tr>
    <tr>
      <td><a href="https://doi.org/10.1038/s41467-021-24435-8">Mapping mutations to the SARS-CoV-2 RBD that escape binding by different classes of antibodies</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_Antibody_Dataset/SARS-CoV-2-RBD_MAP_Rockefeller.csv">SARS-CoV-2-RBD_MAP_Rockefeller.csv</a></td>
    </tr>
    <tr>
      <td><a href="https://doi.org/10.1038/s41586-021-03807-6">SARS-CoV-2 RBD antibodies that maximize breadth and resistance to escape</a></td>
      <td><a href="https://huggingface.co/datasets/xulab-research/MutCleaner/blob/main/RBD_Antibody_Dataset/SARS-CoV-2-RBD_MAP_Vir_mAbs.csv">SARS-CoV-2-RBD_MAP_Vir_mAbs.csv</a></td>
    </tr>
  </tbody>
</table>

The current curated RBD antibody source bundle tracked in this project is
the deduplicated three-table subset: `Moderna`, `Rockefeller`, and
`Vir_mAbs`.


## Core Features

### Sequence Data Manipulation

- **Sequence validation**: Validate DNA, RNA, and protein sequences against predefined alphabets.
- **Sequence transformation**: Support transcription, reverse transcription, translation, and reverse-complement operations.
- **Batch processing**: Process large tabular mutation datasets through reusable cleaning utilities.

### Mutation Analysis

- **Mutation parsing**: Parse amino-acid and codon-level mutation annotations.
- **Mutation inference**: Infer mutation annotations by comparing reference and mutated sequences.
- **Mutation transformation**: Apply mutation annotations to reference sequences and convert codon-level mutations into amino-acid changes.

### Data Cleaning and Preprocessing

- **Standardization**: Harmonize mutation names, sequences, labels, and metadata across heterogeneous datasets.
- **Duplicate handling**: Remove or aggregate redundant mutation records according to dataset-specific rules.
- **Dataset-specific cleaners**: Provide reusable cleaning pipelines for commonly used mutation datasets.

### Pipeline Architecture

- **Modular design**: Compose cleaning workflows from reusable processing components.
- **Parallel processing**: Use multi-core processing for mutation validation, mutation application, and sequence-based mutation inference.
- **Progress tracking**: Monitor long-running cleaning tasks with progress bars and structured execution summaries.

## Examples and Use Cases


## Citation

If you use MutCleaner in your research, please cite:

```bibtex
@software{mutcleaner,
  title={
    MutCleaner: An efficient framework for cleaning, standardizing, and processing biological mutation data.
  },
  author={Ziyu Shi and Yuxiang Tang},
  year={2026},
  url={https://github.com/xulab-research/MutCleaner}
}
```

## License

This project is licensed under the BSD 3-Clause License. See the [LICENSE](LICENSE) file for details.

## Support

- **Issues**: [GitHub Issues](https://github.com/xulab-research/MutCleaner/issues)
- **Discussions**: [GitHub Discussions](https://github.com/xulab-research/MutCleaner/discussions)
