Metadata-Version: 2.1
Name: esat
Version: 2024.0.3
Summary: EPA's Environmental Source Apportionment Toolkit python package.
Author-email: Deron Smith <smith.deron@epa.gov>, Mike Cyterski <cyterski.mike@epa.gov>
Maintainer-email: Deron Smith <smith.deron@epa.gov>
License:                                  Apache License
                                   Version 2.0, January 2004
                                http://www.apache.org/licenses/
        
           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
           1. Definitions.
        
              "License" shall mean the terms and conditions for use, reproduction,
              and distribution as defined by Sections 1 through 9 of this document.
        
              "Licensor" shall mean the copyright owner or entity authorized by
              the copyright owner that is granting the License.
        
              "Legal Entity" shall mean the union of the acting entity and all
              other entities that control, are controlled by, or are under common
              control with that entity. For the purposes of this definition,
              "control" means (i) the power, direct or indirect, to cause the
              direction or management of such entity, whether by contract or
              otherwise, or (ii) ownership of fifty percent (50%) or more of the
              outstanding shares, or (iii) beneficial ownership of such entity.
        
              "You" (or "Your") shall mean an individual or Legal Entity
              exercising permissions granted by this License.
        
              "Source" form shall mean the preferred form for making modifications,
              including but not limited to software source code, documentation
              source, and configuration files.
        
              "Object" form shall mean any form resulting from mechanical
              transformation or translation of a Source form, including but
              not limited to compiled object code, generated documentation,
              and conversions to other media types.
        
              "Work" shall mean the work of authorship, whether in Source or
              Object form, made available under the License, as indicated by a
              copyright notice that is included in or attached to the work
              (an example is provided in the Appendix below).
        
              "Derivative Works" shall mean any work, whether in Source or Object
              form, that is based on (or derived from) the Work and for which the
              editorial revisions, annotations, elaborations, or other modifications
              represent, as a whole, an original work of authorship. For the purposes
              of this License, Derivative Works shall not include works that remain
              separable from, or merely link (or bind by name) to the interfaces of,
              the Work and Derivative Works thereof.
        
              "Contribution" shall mean any work of authorship, including
              the original version of the Work and any modifications or additions
              to that Work or Derivative Works thereof, that is intentionally
              submitted to Licensor for inclusion in the Work by the copyright owner
              or by an individual or Legal Entity authorized to submit on behalf of
              the copyright owner. For the purposes of this definition, "submitted"
              means any form of electronic, verbal, or written communication sent
              to the Licensor or its representatives, including but not limited to
              communication on electronic mailing lists, source code control systems,
              and issue tracking systems that are managed by, or on behalf of, the
              Licensor for the purpose of discussing and improving the Work, but
              excluding communication that is conspicuously marked or otherwise
              designated in writing by the copyright owner as "Not a Contribution."
        
              "Contributor" shall mean Licensor and any individual or Legal Entity
              on behalf of whom a Contribution has been received by Licensor and
              subsequently incorporated within the Work.
        
           2. Grant of Copyright License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              copyright license to reproduce, prepare Derivative Works of,
              publicly display, publicly perform, sublicense, and distribute the
              Work and such Derivative Works in Source or Object form.
        
           3. Grant of Patent License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              (except as stated in this section) patent license to make, have made,
              use, offer to sell, sell, import, and otherwise transfer the Work,
              where such license applies only to those patent claims licensable
              by such Contributor that are necessarily infringed by their
              Contribution(s) alone or by combination of their Contribution(s)
              with the Work to which such Contribution(s) was submitted. If You
              institute patent litigation against any entity (including a
              cross-claim or counterclaim in a lawsuit) alleging that the Work
              or a Contribution incorporated within the Work constitutes direct
              or contributory patent infringement, then any patent licenses
              granted to You under this License for that Work shall terminate
              as of the date such litigation is filed.
        
           4. Redistribution. You may reproduce and distribute copies of the
              Work or Derivative Works thereof in any medium, with or without
              modifications, and in Source or Object form, provided that You
              meet the following conditions:
        
              (a) You must give any other recipients of the Work or
                  Derivative Works a copy of this License; and
        
              (b) You must cause any modified files to carry prominent notices
                  stating that You changed the files; and
        
              (c) You must retain, in the Source form of any Derivative Works
                  that You distribute, all copyright, patent, trademark, and
                  attribution notices from the Source form of the Work,
                  excluding those notices that do not pertain to any part of
                  the Derivative Works; and
        
              (d) If the Work includes a "NOTICE" text file as part of its
                  distribution, then any Derivative Works that You distribute must
                  include a readable copy of the attribution notices contained
                  within such NOTICE file, excluding those notices that do not
                  pertain to any part of the Derivative Works, in at least one
                  of the following places: within a NOTICE text file distributed
                  as part of the Derivative Works; within the Source form or
                  documentation, if provided along with the Derivative Works; or,
                  within a display generated by the Derivative Works, if and
                  wherever such third-party notices normally appear. The contents
                  of the NOTICE file are for informational purposes only and
                  do not modify the License. You may add Your own attribution
                  notices within Derivative Works that You distribute, alongside
                  or as an addendum to the NOTICE text from the Work, provided
                  that such additional attribution notices cannot be construed
                  as modifying the License.
        
              You may add Your own copyright statement to Your modifications and
              may provide additional or different license terms and conditions
              for use, reproduction, or distribution of Your modifications, or
              for any such Derivative Works as a whole, provided Your use,
              reproduction, and distribution of the Work otherwise complies with
              the conditions stated in this License.
        
           5. Submission of Contributions. Unless You explicitly state otherwise,
              any Contribution intentionally submitted for inclusion in the Work
              by You to the Licensor shall be under the terms and conditions of
              this License, without any additional terms or conditions.
              Notwithstanding the above, nothing herein shall supersede or modify
              the terms of any separate license agreement you may have executed
              with Licensor regarding such Contributions.
        
           6. Trademarks. This License does not grant permission to use the trade
              names, trademarks, service marks, or product names of the Licensor,
              except as required for reasonable and customary use in describing the
              origin of the Work and reproducing the content of the NOTICE file.
        
           7. Disclaimer of Warranty. Unless required by applicable law or
              agreed to in writing, Licensor provides the Work (and each
              Contributor provides its Contributions) on an "AS IS" BASIS,
              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
              implied, including, without limitation, any warranties or conditions
              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
              PARTICULAR PURPOSE. You are solely responsible for determining the
              appropriateness of using or redistributing the Work and assume any
              risks associated with Your exercise of permissions under this License.
        
           8. Limitation of Liability. In no event and under no legal theory,
              whether in tort (including negligence), contract, or otherwise,
              unless required by applicable law (such as deliberate and grossly
              negligent acts) or agreed to in writing, shall any Contributor be
              liable to You for damages, including any direct, indirect, special,
              incidental, or consequential damages of any character arising as a
              result of this License or out of the use or inability to use the
              Work (including but not limited to damages for loss of goodwill,
              work stoppage, computer failure or malfunction, or any and all
              other commercial damages or losses), even if such Contributor
              has been advised of the possibility of such damages.
        
           9. Accepting Warranty or Additional Liability. While redistributing
              the Work or Derivative Works thereof, You may choose to offer,
              and charge a fee for, acceptance of support, warranty, indemnity,
              or other liability obligations and/or rights consistent with this
              License. However, in accepting such obligations, You may act only
              on Your own behalf and on Your sole responsibility, not on behalf
              of any other Contributor, and only if You agree to indemnify,
              defend, and hold each Contributor harmless for any liability
              incurred by, or claims asserted against, such Contributor by reason
              of your accepting any such warranty or additional liability.
        
           END OF TERMS AND CONDITIONS
        
           APPENDIX: How to apply the Apache License to your work.
        
              To apply the Apache License to your work, attach the following
              boilerplate notice, with the fields enclosed by brackets "[]"
              replaced with your own identifying information. (Don't include
              the brackets!)  The text should be enclosed in the appropriate
              comment syntax for the file format. We also recommend that a
              file or class name and description of purpose be included on the
              same "printed page" as the copyright notice for easier
              identification within third-party archives.
        
           Copyright 2024 EPA
        
           Licensed under the Apache License, Version 2.0 (the "License");
           you may not use this file except in compliance with the License.
           You may obtain a copy of the License at
        
               http://www.apache.org/licenses/LICENSE-2.0
        
           Unless required by applicable law or agreed to in writing, software
           distributed under the License is distributed on an "AS IS" BASIS,
           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
           See the License for the specific language governing permissions and
           limitations under the License.
        
Project-URL: Documentation, https://quanted.github.io/esat/
Project-URL: Repository, https://github.com/quanted/esat.git
Project-URL: Issues, https://github.com/quanted/esat/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click==8.1.7
Requires-Dist: fuzzy-c-means==1.7.2
Requires-Dist: numpy==1.26.4
Requires-Dist: pandas[excel,performance]==2.2.2
Requires-Dist: plotly==5.19.0
Requires-Dist: scipy==1.12.0
Requires-Dist: tabulate==0.8.10
Requires-Dist: tqdm==4.66.2
Requires-Dist: typer==0.9.4

# Environmental Source Apportionment Toolkit (ESAT)
Last Update: 12-10-2024

## Table of Contents
 - [Description](#description)
 - [Quick Start](#quick-start)
   - [Installation](#installation) 
   - [Example Code](#example-code)
 - [Background](#background)
    - [ESAT Predecessor](#esat-predecessor)
    - [Features](#features)
    - [Notebooks](#notebooks)
 - [Development](#development)
    - [Requirements](#requirements)
    - [Rust Compiling](#rust-compiling)
    - [Creating Docs](#creating-docs)
    - [Building Wheels and Compiling Rust](#building-wheels-and-compiling-rust)
    - [Code Tests](#code-tests)
    - [Community Contributions](#community-contributions)

## Description
The Environmental Source Apportionment Toolkit (ESAT) is an open-source software package that provides API and CLI 
functionality to create source apportionment workflows specifically targeting environmental datasets. Source 
apportionment in environment science is the process of mathematically estimating the profiles and contributions of 
multiple sources in some dataset, and in the case of ESAT, while considering data uncertainty. There are many potential 
use cases for source apportionment in environmental science research, such as in the fields of air quality, water 
quality and potentially many others.

The ESAT toolkit is written in Python and Rust, and uses common packages such as numpy, scipy and pandas for data 
processing. The source apportionment algorithms provided in ESAT include two variants of non-negative matrix 
factorization (NMF), both of which have been written in Rust and contained within the python package. A collection of 
data processing and visualization features are included for data and model analytics. The ESAT package includes a 
synthetic data generator and comparison tools to evaluate ESAT model outputs.

The ESAT python package has been published with the Journal of Open Source Software (JOSS):

[![DOI](https://joss.theoj.org/papers/10.21105/joss.07316/status.svg)](https://doi.org/10.21105/joss.07316)

## Quick Start

ESAT supports python version 3.10, 3.11, and 3.12. As the python package contains compiled code there are OS and python
specific versions, supporting Mac Intel, Mac M1+, Linux, and Windows.

### Documentation
The Python API and CLI documentation can be found at the GitHub ESAT IO site: [https://quanted.github.io/esat/](https://quanted.github.io/esat/)

### Installation
The ESAT python package contains all compiled code and required dependencies and can be installed using pip
```bash
pip install esat
```
which will install the latest version that supports and is available for your python version and OS.

Development versions of ESAT can be found on the GitHub actions page, for logged-in users, 
under the 'Build and Publish Wheel' workflow. The latest version of the package will be available as an artifact for 
download in the 'Artifacts' section of the completed workflow. There wheel files can be found for specific versions
of python and supported operating systems. 

If an error message appears during installation stating that the 
package is not supported check that the correct OS and python version are being installed for that system. The python 
wheels can be installed directly using 
```bash
pip install <wheel file name>
```
The esat python package is recommended to be installed in its own dedicated python virtual environment or conda environment.

To run the jupyter notebooks, install jupyterlab into the esat python environmental
```bash
pip install jupyterlab
```

### Example Code
Jupyter notebooks containing complete code examples, using sample datasets, are available for the 
[source apportionment workflow](notebooks/epa_esat_workflow_01.ipynb) and the [simulator workflow](notebooks/epa_esat_simulator_01.ipynb). 

## Background

### ESAT Predecessor
A widely used application used for environmental source apportionment is the EPA's Positive Matrix Factorization version
5 (PMF5), which is a broadly used tool with an international user community. The PMF5 application is a mathematical tool
that processes a dataset of feature concentrations across many samples (and concentration uncertainties) to estimate a 
set of source profiles and their contributions. PMF5 can be used on a wide range of environmental data and is a powerful
tool for estimating source apportionment: 
https://www.epa.gov/air-research/positive-matrix-factorization-model-environmental-data-analyses

PMF5 was released in 2014 and is no longer being supported. The math engine used in PMF5 is proprietary and the source
code has not been made public. One of the primary purposes of ESAT was to recreate the source apportionment workflow and
mathematics as an open-source software package to offer a modernized option for environmental source apportionment. 
Other reasons for developing ESAT was to offer increased maintainability, development efficient, thorough documentation,
modern optimizations, new features and customized workflows for novel use cases.

### Features
ESAT python package focuses on source apportionment estimates using NMF algorithms. These
algorithms are implemented both in python using numpy functions and in Rust (default) for an optimization option. The
two currently available are:
 1. LS-NMF: Least-squares NMF, a well documented and widely uses NMF algorithm. The ls-nmf algorithm is available in the NMF R package.

 2. WS-NMF: Weight-Semi NMF, a variant of the NMF algorithm which allows for negative values in both the input data matrix and in the factor contribution matrix.

Source apportionment solution error estimation methods are also available, which are the same methods that are found in PMF5.
These are:
 1. Bootstrap (BS): the input dataset is divided into multiple blocks and randomly reassembled to attempt to quantify the variability in the factor profiles and contributions.

 2. Displacement (DISP): the solution factor profiles are all individually shifted (both increased and decreased) to determine the amount of change required for the loss value to reach specific dQ (change in Q) values.

 3. Bootstrap-Displacement (BS-DISP): the combination of the two error estimation methods. Where for each Bootstrap dataset/model, all or targeted factor profile values are adjusted using the DISP method.

ESAT includes constrained models, as found in PMF5, where selecting a source apportionment model there is the option to add constraints through defining specific value constraints or define value correlations as a collection of linear equations.

Lastly, ESAT includes a data simulator which allows for random or use defined synthetic source profiles and contributions to be used in ESAT to evaluating how well the original synthetic data can be recreated.

### Limitations
Matrix factorization algorithms are fall under the group of optimization, or minimization, algorithms which attempt to 
find a minima based upon some loss function and stopping condition. These algorithms are classified as NP-Complete, a 
category of algorithms which are nondeterministic polynomial time complete and there is no known way to quickly find a 
solution. Given this limitation of NMF, a solution can only be considered a local minima with no known way to guarantee 
or prove it is the globally optimal solution. One approach for helping determine that a solution is a good solution is 
by producing many such solutions with a constricted convergence criteria or stopping condition. Then evaluating these 
solutions to determine which, if any, correspond to the best actual representation or model of the data given domain 
knowledge and expertise. 

NMF algorithms are data-agnostic, operates the same on any correctly structured data regardless of domain, potential 
leading to another limitation, interpretation of the solution. An important component in evaluating whether or not to 
use ESAT, or any NMF algorithm, on a dataset is to determine how to interpret factor profiles and contributions. How 
this is done is fully dependent on the data and domain of the input dataset, such as units, types of features, temporal 
or spatial considerations, etc. 


### Notebooks
Juypter notebooks are available that demonstrate the complete source apportionment and error estimation workflow found in PMF5, demonstrated in notebooks/epa_esat_workflow_01.ipynb

The simulator notebook provides examples for creating the synthetic profiles and contributions dataset and using the evaluation features to see how 'well' ESAT can recreate those profiles and contributions.

Other notebooks are included which were used during development and verifying visualizations. 

## Development
### Requirements
* Core ESAT python package requirements can be found in the requirements.txt file.
* The python requirements for creating the code documentation can be found in the doc-requirements.txt file.
* Full development python package requirements can be found in the _dev-requirements.txt file (not actively maintained).

The ESAT python codebase includes github workflow actions which run:
1. Run python build to compile the Rust code and create python packages for python 3.10, 3.11 and 3.12 on Linux, Windows and MacOS.
2. Recreate code documentation from the README.md file, code docstrings for the Python API and CLI. Documentation is used to update the github documentation site for ESAT.

### Rust Compiling
The python package includes a Rust module for running the algorithm update procedures, which requires local compiling to execute.

To run the Rust functions that is specified by the optimized parameter, requires that Rust is installed (https://www.rust-lang.org/tools/install) and the python package maturin (https://pypi.org/project/maturin/) is installed to the python development environment. 
Then from the python env and the project root, executing <i>maturin develop</i> will compile the Rust code and place it in your python environment path. To compile optimized Rust code, include the <i>-r</i> or <i>--release</i> tags.

The Rust code can also be compiled to the target directory inside of project root using <i>maturin build</i>.

The rust functions are imported as python functions, with the 'from esat_rust import esat_rust'.

When creating the python package the pyproject.toml specifies that both setuptools and setuptool-rust are used. Setuptools-rust is required for compiling the Rust code during package build. 

### Creating Docs
The documentation is created using sphinx and several extensions.

To create or add new rst files run <i>sphinx-apidoc -o docs esat</i>.

To create, update or modify the existing documentation html, run 
<i>sphinx-build -M html . docs</i> from the command line at the project root directory running the python environment with the necessary sphinx packages.

### Building Wheels and Compiling Rust
The ESAT python package and cli are built using setuptools and setuptools-rust, with configuration details defined in pyproject.toml and Cargo.toml. 

The python package can be built with the standard <i>python -m build</i> from the project root directory. 

Build will compile the rust code and package up the python
code combining them into the wheel for distribution. The resulting wheel with the compiled code is available on github as a workflow artifact for the targeted architecture and python version.

The python package will be available on pypi.org in the near future.

### Code Tests
A collection of pytest tests have been created to test functionality, mostly as systems tests, which can be executed as 

<i>coverage run -m pytest tests</i>

with the coverage results displayed by 

<i>coverage report</i>

While the overall coverage percentage is low, the majority of the untested code is for visualization functions with all 
core functionality covered by tests.

### Community Contributions
For those in the user community wishing to contribute to this project:
 * Code updates can be made through pull requests that will be reviewed by repository maintainers.
 * Software, code, or algorithm related bugs and issues can be submitted directly as issues on the GitHub repository.
 * Support can be requested through GitHub issues or through email at [esat@epa.gov](mailto:esat@epa.gov).

## Disclaimer 
ESAT development has been funded by U.S. EPA.  Mention of any trade names, products, or services does not convey, and 
should not be interpreted as conveying, official EPA approval, endorsement, or recommendation. The views expressed in 
this README are those of the authors and do not necessarily represent the views or policies of the US EPA.
