Metadata-Version: 2.4
Name: miles-guess
Version: 2025.1.0
Author-email: University Corporation for Atmospheric Research <milescore@ucar.edu>
License: Creative Commons Legal Code
        
        CC0 1.0 Universal
        
            CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
            LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
            ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
            INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
            REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
            PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
            THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
            HEREUNDER.
        
        Statement of Purpose
        
        The laws of most jurisdictions throughout the world automatically confer
        exclusive Copyright and Related Rights (defined below) upon the creator
        and subsequent owner(s) (each and all, an "owner") of an original work of
        authorship and/or a database (each, a "Work").
        
        Certain owners wish to permanently relinquish those rights to a Work for
        the purpose of contributing to a commons of creative, cultural and
        scientific works ("Commons") that the public can reliably and without fear
        of later claims of infringement build upon, modify, incorporate in other
        works, reuse and redistribute as freely as possible in any form whatsoever
        and for any purposes, including without limitation commercial purposes.
        These owners may contribute to the Commons to promote the ideal of a free
        culture and the further production of creative, cultural and scientific
        works, or to gain reputation or greater distribution for their Work in
        part through the use and efforts of others.
        
        For these and/or other purposes and motivations, and without any
        expectation of additional consideration or compensation, the person
        associating CC0 with a Work (the "Affirmer"), to the extent that he or she
        is an owner of Copyright and Related Rights in the Work, voluntarily
        elects to apply CC0 to the Work and publicly distribute the Work under its
        terms, with knowledge of his or her Copyright and Related Rights in the
        Work and the meaning and intended legal effect of CC0 on those rights.
        
        1. Copyright and Related Rights. A Work made available under CC0 may be
        protected by copyright and related or neighboring rights ("Copyright and
        Related Rights"). Copyright and Related Rights include, but are not
        limited to, the following:
        
          i. the right to reproduce, adapt, distribute, perform, display,
             communicate, and translate a Work;
         ii. moral rights retained by the original author(s) and/or performer(s);
        iii. publicity and privacy rights pertaining to a person's image or
             likeness depicted in a Work;
         iv. rights protecting against unfair competition in regards to a Work,
             subject to the limitations in paragraph 4(a), below;
          v. rights protecting the extraction, dissemination, use and reuse of data
             in a Work;
         vi. database rights (such as those arising under Directive 96/9/EC of the
             European Parliament and of the Council of 11 March 1996 on the legal
             protection of databases, and under any national implementation
             thereof, including any amended or successor version of such
             directive); and
        vii. other similar, equivalent or corresponding rights throughout the
             world based on applicable law or treaty, and any national
             implementations thereof.
        
        2. Waiver. To the greatest extent permitted by, but not in contravention
        of, applicable law, Affirmer hereby overtly, fully, permanently,
        irrevocably and unconditionally waives, abandons, and surrenders all of
        Affirmer's Copyright and Related Rights and associated claims and causes
        of action, whether now known or unknown (including existing as well as
        future claims and causes of action), in the Work (i) in all territories
        worldwide, (ii) for the maximum duration provided by applicable law or
        treaty (including future time extensions), (iii) in any current or future
        medium and for any number of copies, and (iv) for any purpose whatsoever,
        including without limitation commercial, advertising or promotional
        purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
        member of the public at large and to the detriment of Affirmer's heirs and
        successors, fully intending that such Waiver shall not be subject to
        revocation, rescission, cancellation, termination, or any other legal or
        equitable action to disrupt the quiet enjoyment of the Work by the public
        as contemplated by Affirmer's express Statement of Purpose.
        
        3. Public License Fallback. Should any part of the Waiver for any reason
        be judged legally invalid or ineffective under applicable law, then the
        Waiver shall be preserved to the maximum extent permitted taking into
        account Affirmer's express Statement of Purpose. In addition, to the
        extent the Waiver is so judged Affirmer hereby grants to each affected
        person a royalty-free, non transferable, non sublicensable, non exclusive,
        irrevocable and unconditional license to exercise Affirmer's Copyright and
        Related Rights in the Work (i) in all territories worldwide, (ii) for the
        maximum duration provided by applicable law or treaty (including future
        time extensions), (iii) in any current or future medium and for any number
        of copies, and (iv) for any purpose whatsoever, including without
        limitation commercial, advertising or promotional purposes (the
        "License"). The License shall be deemed effective as of the date CC0 was
        applied by Affirmer to the Work. Should any part of the License for any
        reason be judged legally invalid or ineffective under applicable law, such
        partial invalidity or ineffectiveness shall not invalidate the remainder
        of the License, and in such case Affirmer hereby affirms that he or she
        will not (i) exercise any of his or her remaining Copyright and Related
        Rights in the Work or (ii) assert any associated claims and causes of
        action with respect to the Work, in either case contrary to Affirmer's
        express Statement of Purpose.
        
        4. Limitations and Disclaimers.
        
         a. No trademark or patent rights held by Affirmer are waived, abandoned,
            surrendered, licensed or otherwise affected by this document.
         b. Affirmer offers the Work as-is and makes no representations or
            warranties of any kind concerning the Work, express, implied,
            statutory or otherwise, including without limitation warranties of
            title, merchantability, fitness for a particular purpose, non
            infringement, or the absence of latent or other defects, accuracy, or
            the present or absence of errors, whether or not discoverable, all to
            the greatest extent permissible under applicable law.
         c. Affirmer disclaims responsibility for clearing rights of other persons
            that may apply to the Work or any use thereof, including without
            limitation any person's Copyright and Related Rights in the Work.
            Further, Affirmer disclaims responsibility for obtaining any necessary
            consents, permissions or other rights required for any use of the
            Work.
         d. Affirmer understands and acknowledges that Creative Commons is not a
            party to this document and has no duty or obligation with respect to
            this CC0 or use of the Work.
        
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: matplotlib
Requires-Dist: pandas
Requires-Dist: xarray
Requires-Dist: keras
Requires-Dist: scikit-learn
Requires-Dist: netcdf4
Requires-Dist: pyyaml
Requires-Dist: tqdm
Requires-Dist: sphinx
Requires-Dist: numba
Requires-Dist: properscoring
Requires-Dist: pyarrow
Requires-Dist: imbalanced-learn
Requires-Dist: bridgescaler
Requires-Dist: echo-opt
Requires-Dist: hagelslag
Requires-Dist: jupyter
Requires-Dist: nbsphinx
Requires-Dist: myst_parser
Requires-Dist: sphinx_book_theme
Requires-Dist: hagelslag
Provides-Extra: tensorflow
Requires-Dist: tensorflow>=2.16.0; extra == "tensorflow"
Provides-Extra: tensorflow-gpu
Requires-Dist: tensorflow[and-cuda]>=2.16.0; extra == "tensorflow-gpu"
Provides-Extra: torch
Requires-Dist: torch; extra == "torch"
Requires-Dist: safetensors; extra == "torch"
Provides-Extra: jax
Requires-Dist: jax; extra == "jax"
Dynamic: license-file

# MILES-Guess
Generalized Uncertainty for Earth System Science (GUESS)

Developed by the Machine Ingetration and Learning for Earth Systems (MILES) group at the NSF National Center for Atmospheric Research (NCAR), Boulder CO, USA

## Contributors 
* John Schreck
* David John Gagne
* Charlie Becker
* Gabrielle Gantos
* Dhamma Kimpara
* Thomas Martin

## Documentation
Full documentation is [here](https://miles-guess.readthedocs.io/en/latest/).

## Quick Setup

Install in your Python environment with the following command:
```bash
pip install miles-guess
```
If you want to install a particular backend (tensorflow, tensorflow_gpu, torch, jax): 
```bash
pip install miles-guess[<backend>]
```
## Setup from Scratch

Install the Miniconda Python installer available
[here](https://docs.conda.io/en/latest/miniconda.html).

First clone the miles-guess repo from github.
```bash
git clone https://github.com/ai2es/miles-guess.git`
cd miles-guess
```

Create a conda environment for non-Casper/Derecho users:
```bash
mamba env create -f environment.yml`
conda activate guess`
```

Create a conda environment for Casper/Derecho users including Tensorflow 2.15 with GPU support.
```bash
mamba env create -f environment_gpu.yml`
conda activate guess
```

## Using miles-guess

The law of total variance for each model prediction target may be computed as

$$LoTV = E[\sigma^2] + Var[\mu]$$ 

which is the sum of aleatoric and epistemic contributions, respectively. The MILES-GUESS package contains options for using either Keras or PyTorch for computing quantites according to the LoTV as well as utilizing Dempster-Shafer theory uncertainty in the classifier case. 

For detailed information about training with Keras, refer to [the Keras training details README](docs/source/keras.md). There three scripts for training three regression models, and one for training categorical models. The regression examples are trained on our surface layer ("SL") dataset for predicting latent heat and other quantities, 
and the categorical example is trained on a precipitation dataset ("p-type").

For pyTorch, please visit the [the pyTorch training details README](docs/source/torch.md) where details on training scripts for both evidential standard classification tasks are detailed. Torch examples use the same datasets as the Keras models. The torch training code will also scale on GPUs, and is compatitible with DDP and FSDP.

<!--
### 1a. Train/evaluate a deterministic multi-layer perceptrion (MLP) on the SL dataset:
```bash
python3 applications/train_mlp_SL.py -c config/model_mlp_SL.yml
```

### 1b. Train/evaluate a parametric "Gaussian" MLP on the SL dataset:
```bash

python applications/train_gaussian_SL.py -c config/model_gaussian_SL.yml
```

### 1c. Train/evaluate a parametric "normal-inverse gamma" (evidential) MLP on the SL dataset:
```bash
python applications/train_evidential_SL.py -c config/model_evidential_SL.yml
```

### 2a. Train a categorical MLP classifier on the p-type dataset:
```bash
python applications/train_classifier_ptype.py -c config/model_classifier_ptype.yml
```

### 2b. Train an evidential MLP classifier on the p-type dataset:
```bash
python applications/train_classifier_ptype.py -c config/model_evidential_ptype.yml
```

### 2c. Evaluate a categorical/evidential classifier on the p-type dataset:
```bash
python applications/evaluate_ptype.py -c config/model_classifier_ptype.yml
```


## 3. Ensembling modes for the deterministic model (1a)

There are four "modes" for training the deterministic MLP (1a) that are controlled using the "ensemble" field in a model configuration.
```yaml
ensemble:
    n_models: 100
    n_splits: 20
    monte_carlo_passes: 0
```
where n_models means the number of models to train using a fixed data split with variable initial weight initializations, n_splits means the number of models to train using variable training and validation splits (random initializations), and mc_carlo_passes means the number of MC-dropout evaluations performed on a given input to the model. 

### 3a. Single Mode
```yaml
ensemble:
    n_models: 1
    n_splits: 1
    monte_carlo_passes: 0
```
Train a single deterministic model (no uncertainty evaluation). If MC passes > 0, an ensemble is created after the model finishes training.

### 3b. Data Mode
```yaml
ensemble:
    n_models: 1
    n_splits: 10
    monte_carlo_passes: 100
```
Create an ensemble of models (random initialization) using cross-validation splits. If MC passes > 0, an ensemble is created after each model finishes training on the test holdout. The LOTV may then be applied to the ensemble created from cross-validation. Otherwise a single ensemble is created but the LOTV is not applied. 

### 3c. Model Mode
```yaml
ensemble:
    n_models: 10
    n_splits: 1
    monte_carlo_passes: 100
```
Create an ensemble of models using a fixed train/validation/test data split and variable model layer weight initializations. If MC passes > 0, an ensemble is created after each model finishes training. The LOTV may then be applied to the ensemble created from variable weight initializations to obtain uncertainty estimations for each prediction target. Otherwise a single ensemble is created but the LOTV is not applied. 

### 3d. Ensemble Mode
```yaml
ensemble:
    n_models: 1
    n_splits: 1
    monte_carlo_passes: 0
```
Create an ensemble of ensembles. The first ensemble is created using cross validation and a fixed weight initialization, from which a mean and variance may be obtained for each prediction target. The second ensemble is created by varying the weight initalization that can then be used with the LOTV to obtain uncertainty estimations for each prediction target. The MC steps field is ignored in ensemble mode. 

## 4. Ensembling modes for the Gaussian parametric model (1b)
There are three "modes" for training the Gaussian MLP (1b).

### 4a. Single Mode
```yaml
ensemble:
    n_models: 1
    n_splits: 1
    monte_carlo_passes: 0
```
Train a single deterministic model (no LOTV evaluation). If MC passes > 0, an ensemble is created after the model finishes training (LOTV evaluation).

### 4b. Data Mode
```yaml
ensemble:
    n_models: 1
    n_splits: 10
    monte_carlo_passes: 0
```
Create an ensemble of models using cross-validation splits, and then LOTV evaluation.

### 4c. Model Mode
```yaml
ensemble:
    n_models: 10
    n_splits: 1
    monte_carlo_passes: 0
```
Create an ensemble of models using different random initializations and a fixed cross-validation split, and then LOTV evaluation.

## Configuration files
In addition to the ensemble field, the other fields in the configuration file are 

```yaml
seed: 1000
save_loc: "/path/to/save/directory"
training_metric: "val_mae"
direction: "min"
```

where seed allows for reproducability, save_loc is where data will be saved, and training metric and direction are used as the validation metric (and direction).

For regression tasks, other fields in a configuration file are model and callbacks:
```yaml

model:
    activation: relu
    batch_size: 193
    dropout_alpha: 0.2
    epochs: 200
    evidential_coef: 0.6654439861214466
    hidden_layers: 3
    hidden_neurons: 6088
    kernel_reg: l2
    l1_weight: 0.0
    l2_weight: 7.908676527243475e-10
    lr: 3.5779279071474884e-05
    metrics: mae
    optimizer: adam
    uncertainties: true
    use_dropout: true
    use_noise: false
    verbose: 2
      
callbacks:
  EarlyStopping:
    monitor: "val_loss"
    patience: 5
    mode: "min"
    verbose: 0
  ReduceLROnPlateau: 
    monitor: "val_loss"
    factor: 0.1
    patience: 2
    min_lr: 1.0e-12
    min_delta: 1.0e-08
    mode: "min"
    verbose: 0
  CSVLogger:
    filename: "training_log.csv"
    separator: ","
    append: False
  ModelCheckpoint:
    filepath: "model.h5"
    monitor: "val_loss"
    save_weights: True
    save_best_only: True
    mode: "min"
    verbose: 0
```

For categorical tasks, the model field changes slightly:

```yaml
model:
    activation: leaky
    balanced_classes: 1
    batch_size: 3097
    dropout_alpha: 0.31256692323263807
    epochs: 200
    hidden_layers: 4
    hidden_neurons: 6024
    loss: categorical_crossentropy
    loss_weights:
    - 21.465788717561477
    - 83.31367732936326
    - 136.50944842077058
    - 152.62042204485107
    lr: 0.0004035503144482269
    optimizer: adam
    output_activation: softmax
    use_dropout: 1
    verbose: 0
```
where the user has two options:

(1) A standard deterministic classifier is trained when 
```yaml 
    loss: cateogical_crossentropy
    output_activation: softmax
```
(2) An evidential classifier is trained when 
```yaml 
    loss: dirichlet
    output_activation: linear
```

Callbacks are not required in regression training, however a custom callback which tracks the current epoch is requried for the categorical model and is added automatically (the user does not need to specify it in the callbacks field). The user may add any other supported keras callback by adding the relevant fields to the callbacks field. 

Depending on the problem, a data field is customized and also present in the configuration files. See the examples for surface layer and p-type data sets for more details. 


## ECHO hyperparameter optimization 

Configuration files are also supplied for use with the Earth Computing Hyperparameter Optimization (ECHO) package. See the echo package https://github.com/NCAR/echo-opt/tree/main/echo for more details on the configuration fields. -->
