Metadata-Version: 2.4
Name: smftools
Version: 2.18.0
Summary: Single Molecule Footprinting Analysis in Python.
Project-URL: Source, https://github.com/jkmckenna/smftools
Project-URL: Documentation, https://smftools.readthedocs.io/
Author: Joseph McKenna
Maintainer-email: Joseph McKenna <jkmckenna@berkeley.edu>
License: MIT License
        
        Copyright (c) 2024 jkmckenna
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: anndata,chromatin-accessibility,machine-learning,nanopore,protein-dna-binding,single-locus,single-molecule-footprinting
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.11
Requires-Dist: anndata<0.13,>=0.12
Requires-Dist: biopython>=1.79
Requires-Dist: click
Requires-Dist: dask[array]
Requires-Dist: igraph
Requires-Dist: joblib
Requires-Dist: leidenalg
Requires-Dist: matplotlib>=3.6.2
Requires-Dist: numpy<2,>=1.22.0
Requires-Dist: pandas<3,>=2.1
Requires-Dist: psutil>=5.9.0
Requires-Dist: pyarrow>=15
Requires-Dist: pysam>=0.19.1
Requires-Dist: scikit-learn>=1.2
Requires-Dist: scipy>=1.7.3
Requires-Dist: seaborn>=0.11
Requires-Dist: tensorly
Requires-Dist: torch>=2.0
Requires-Dist: tqdm
Requires-Dist: umap-learn>=0.5.5
Requires-Dist: zarr<4,>=3.1
Provides-Extra: all
Requires-Dist: captum; extra == 'all'
Requires-Dist: duckdb; extra == 'all'
Requires-Dist: edlib; extra == 'all'
Requires-Dist: hydra-core; extra == 'all'
Requires-Dist: igraph; extra == 'all'
Requires-Dist: leidenalg; extra == 'all'
Requires-Dist: lightning; extra == 'all'
Requires-Dist: multiqc; extra == 'all'
Requires-Dist: networkx>=3.2; extra == 'all'
Requires-Dist: omegaconf; extra == 'all'
Requires-Dist: pod5>=0.1.21; extra == 'all'
Requires-Dist: pybedtools>=0.12.0; extra == 'all'
Requires-Dist: pybigwig>=0.3.24; extra == 'all'
Requires-Dist: pynndescent>=0.5.10; extra == 'all'
Requires-Dist: shap; extra == 'all'
Requires-Dist: tensorly; extra == 'all'
Requires-Dist: umap-learn>=0.5.5; extra == 'all'
Requires-Dist: upsetplot; extra == 'all'
Requires-Dist: wandb; extra == 'all'
Requires-Dist: xarray; extra == 'all'
Requires-Dist: xgboost>=1.7; extra == 'all'
Provides-Extra: all-2
Requires-Dist: captum; extra == 'all-2'
Requires-Dist: duckdb; extra == 'all-2'
Requires-Dist: edlib; extra == 'all-2'
Requires-Dist: hydra-core; extra == 'all-2'
Requires-Dist: igraph; extra == 'all-2'
Requires-Dist: leidenalg; extra == 'all-2'
Requires-Dist: lightning; extra == 'all-2'
Requires-Dist: multiqc; extra == 'all-2'
Requires-Dist: networkx>=3.2; extra == 'all-2'
Requires-Dist: omegaconf; extra == 'all-2'
Requires-Dist: pod5>=0.1.21; extra == 'all-2'
Requires-Dist: pybedtools>=0.12.0; extra == 'all-2'
Requires-Dist: pybigwig>=0.3.24; extra == 'all-2'
Requires-Dist: pynndescent>=0.5.10; extra == 'all-2'
Requires-Dist: shap; extra == 'all-2'
Requires-Dist: tensorly; extra == 'all-2'
Requires-Dist: umap-learn>=0.5.5; extra == 'all-2'
Requires-Dist: upsetplot; extra == 'all-2'
Requires-Dist: wandb; extra == 'all-2'
Requires-Dist: xarray; extra == 'all-2'
Requires-Dist: xgboost>=1.7; extra == 'all-2'
Provides-Extra: analysis
Requires-Dist: igraph; extra == 'analysis'
Requires-Dist: leidenalg; extra == 'analysis'
Requires-Dist: networkx>=3.2; extra == 'analysis'
Requires-Dist: pynndescent>=0.5.10; extra == 'analysis'
Requires-Dist: tensorly; extra == 'analysis'
Requires-Dist: umap-learn>=0.5.5; extra == 'analysis'
Requires-Dist: upsetplot; extra == 'analysis'
Requires-Dist: xgboost>=1.7; extra == 'analysis'
Provides-Extra: catalog
Requires-Dist: duckdb; extra == 'catalog'
Provides-Extra: cluster
Requires-Dist: igraph; extra == 'cluster'
Requires-Dist: leidenalg; extra == 'cluster'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: hatch; extra == 'dev'
Requires-Dist: hatch-vcs; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff==0.15.0; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: docs
Requires-Dist: ipython>=7.20; extra == 'docs'
Requires-Dist: matplotlib!=3.6.1; extra == 'docs'
Requires-Dist: myst-nb<2,>=1; extra == 'docs'
Requires-Dist: myst-parser<3,>=2; extra == 'docs'
Requires-Dist: nbsphinx>=0.9; extra == 'docs'
Requires-Dist: pyyaml; extra == 'docs'
Requires-Dist: readthedocs-sphinx-search; extra == 'docs'
Requires-Dist: setuptools; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints<4,>=1.25.2; extra == 'docs'
Requires-Dist: sphinx-book-theme<2,>=1.1; extra == 'docs'
Requires-Dist: sphinx-click<7,>=5; extra == 'docs'
Requires-Dist: sphinx-copybutton<0.6,>=0.5; extra == 'docs'
Requires-Dist: sphinx-design; extra == 'docs'
Requires-Dist: sphinx<8,>=7; extra == 'docs'
Requires-Dist: sphinxcontrib-bibtex<3,>=2; extra == 'docs'
Requires-Dist: sphinxext-opengraph<0.10,>=0.9; extra == 'docs'
Provides-Extra: genome-io
Requires-Dist: pybedtools>=0.12.0; extra == 'genome-io'
Requires-Dist: pybigwig>=0.3.24; extra == 'genome-io'
Provides-Extra: lazy
Requires-Dist: xarray; extra == 'lazy'
Provides-Extra: misc
Requires-Dist: networkx>=3.2; extra == 'misc'
Requires-Dist: upsetplot; extra == 'misc'
Provides-Extra: ml-base
Requires-Dist: scikit-learn>=1.2; extra == 'ml-base'
Requires-Dist: tensorly; extra == 'ml-base'
Requires-Dist: torch>=2.0; extra == 'ml-base'
Requires-Dist: xgboost>=1.7; extra == 'ml-base'
Provides-Extra: ml-extended
Requires-Dist: captum; extra == 'ml-extended'
Requires-Dist: hydra-core; extra == 'ml-extended'
Requires-Dist: lightning; extra == 'ml-extended'
Requires-Dist: omegaconf; extra == 'ml-extended'
Requires-Dist: shap; extra == 'ml-extended'
Requires-Dist: wandb; extra == 'ml-extended'
Provides-Extra: ont
Requires-Dist: pod5>=0.1.21; extra == 'ont'
Provides-Extra: plotting
Requires-Dist: matplotlib>=3.6.2; extra == 'plotting'
Requires-Dist: seaborn>=0.11; extra == 'plotting'
Provides-Extra: project
Requires-Dist: duckdb; extra == 'project'
Requires-Dist: xarray; extra == 'project'
Provides-Extra: pybedtools
Requires-Dist: pybedtools>=0.12.0; extra == 'pybedtools'
Provides-Extra: pybigwig
Requires-Dist: pybigwig>=0.3.24; extra == 'pybigwig'
Provides-Extra: pysam
Requires-Dist: pysam>=0.19.1; extra == 'pysam'
Provides-Extra: qc
Requires-Dist: multiqc; extra == 'qc'
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == 'torch'
Provides-Extra: umap
Requires-Dist: pynndescent>=0.5.10; extra == 'umap'
Requires-Dist: umap-learn>=0.5.5; extra == 'umap'
Provides-Extra: umi
Requires-Dist: edlib; extra == 'umi'
Provides-Extra: xgboost
Requires-Dist: xgboost>=1.7; extra == 'xgboost'
Description-Content-Type: text/markdown

[![PyPI](https://img.shields.io/pypi/v/smftools.svg)](https://pypi.org/project/smftools)
[![Docs](https://readthedocs.org/projects/smftools/badge/?version=latest)](https://smftools.readthedocs.io/en/latest/?badge=latest)

# smftools
A Python tool for automated processing of raw sequencing data derived from single molecule footprinting experiments into [zarr](https://github.com/zarr-developers/zarr-python) and [parquet](https://github.com/apache/parquet-format/) data formats. Experimental data is organized into projects using [DuckDB](https://github.com/duckdb/duckdb) (optional; falls back to a pandas/pyarrow union without it) to facilitate growth of data collections for scientific projects. An additional analysis subpackage provides functionality that can be imported for custom analyses and interactive analysis sessions.

## Philosophy
While genomic data structures (SAM/BAM) were built to store read alignment data and basic read metadata along large references, integration of downstream analyses is not feasible using this format alone. Smftools integrates experimental analyses across file formats, linking raw sequencing data files, BAM alignment files, and downstream analyses into modern storage formats such as zarr for arrays and parquet for tables. This enables efficient partitioned data storage, rapid and parallel data operations, hierarchical metadata handling, and seamless integration with machine-learning workflows. Furthermore, functionality is modularized into multiple processing stages, enabling analysis to restart from convenient checkpoints without having to rerun the full workflow. Collections of experiments are managed under smftools projects, which indexes individual experiments and combine them for continuously growing scientific projects.

## Installation
SMFtools requires Python 3.11 or newer. The default installation supports
`smftools experiment full` from a basecalled BAM, including the portable pysam
BAM backend, preprocessing, spatial analysis, HMM analysis, and plotting:

```bash
pip install smftools

# Or install the current development checkout.
git clone https://github.com/jkmckenna/smftools.git
cd smftools
pip install -e .
```

Install only the optional capabilities a run needs, for example
`pip install -e ".[ont,project]"`:

- `ont` -> POD5 input and Nanopore signal I/O.
- `umi` -> edit-distance-based UMI and barcode processing.
- `genome-io` -> pybedtools and pyBigWig genome-format backends.
- `project` -> DuckDB catalogs and lazy xarray-backed project reads.
- `analysis` -> downstream clustering, UMAP, tensor, graph, and XGBoost analyses.
- `ml-extended` -> Captum, Lightning, SHAP, Weights & Biases, and related ML tools.
- `qc` -> MultiQC report generation.
- `all` -> every optional runtime capability.

Older fine-grained extras remain compatibility aliases. In particular, `torch`,
`plotting`, and `pysam` are now redundant because those dependencies are part of
the default workflow install, and `all_2` is an alias for `all`.

Canonical contributor installs use dependency groups so test, lint, and docs
tools are never installed by a normal runtime install:

```bash
python -m pip install -e ".[all]"
python -m pip install --group dev --group docs
```

See the [installation guide](https://smftools.readthedocs.io/en/latest/installation.html)
for profile details and external command-line requirements.

## Command-line interface
smftools exposes two top-level command groups (`smftools --help` for the full list):

**`smftools experiment <config_path>`** -> pipeline stages for a single experiment:
- `raw` -> Prepare BAM artifacts and write the ragged raw store.
- `load` -> Optionally pre-build the dense zarr cache from raw artifacts.
- `preprocess` -> QC, filtering, and read-level preprocessing.
- `spatial` -> Spatial signal analysis.
- `hmm` -> HMM feature annotation and plotting.
- `full` -> Composed workflow: raw, preprocess, spatial, hmm.
- `batch` -> Run any single stage across many experiments listed in a CSV/TSV/TXT file.
- `concatenate`, `export-fastq`, `plot-current` -> supporting utilities.

**`smftools project <project_dir>`** -> registering and querying across experiments:
- `init` -> Initialize a project directory + registry.
- `add` / `remove` -> Register or deactivate an experiment in the project.
- `list` -> List registered experiments and harmonized references.
- `materialize` -> Pool a reference across matching experiments into one AnnData.
- `sample-store-list` -> List cataloged per-sample-store partitions.
- `export-fastq` -> Write one FASTQ per barcode of QC-passed reads, across every registered experiment.

Full documentation for each command and its options is at [smftools.readthedocs.io](https://smftools.readthedocs.io/).

## Dependencies
The following CLI tools need to be installed and configured before using the informatics (smftools.inform) module of smftools, which is used by the `smftools experiment raw` CLI command:
1) [Dorado](https://github.com/nanoporetech/dorado) -> Basecalling, alignment, demultiplexing. Required for Nanopore SMF experiments, but not Illumina SMF experiments.
2) [Minimap2](https://github.com/lh3/minimap2) -> Aligner if not using dorado. Support for other aligners could eventually be added if needed.
3) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting read level methylation metrics from the MM/ML tags in BAM files. Only required for direct modification detection SMF protocols.

## License
MIT -- see [LICENSE](LICENSE).
