Metadata-Version: 2.4
Name: mutts
Version: 2.0.0
Summary: Metadata for User facility Template Transformations
License: MIT
Keywords: NMDC,US DOE user facilities,metadata translation
Author: Sujay Patil
Author-email: spatil@lbl.gov
Requires-Python: >=3.12,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: click (>=8.1.7,<9.0.0)
Requires-Dist: openpyxl (>=3.1.0,<4.0.0)
Requires-Dist: pandas (>=2.2.0,<3.0.0)
Requires-Dist: python-dotenv (>=1.0.0,<2.0.0)
Requires-Dist: requests (>=2.32.0,<3.0.0)
Project-URL: Homepage, https://github.com/microbiomedata/metadata-for-user-facility-template-transformations
Project-URL: Repository, https://github.com/microbiomedata/metadata-for-user-facility-template-transformations
Description-Content-Type: text/markdown

# Metadata for User facility Template Transformations (MUTTs)

## Table of Contents
- [Metadata for User facility Template Transformations (MUTTs)](#metadata-for-user-facility-template-transformations-mutts)
  - [Table of Contents](#table-of-contents)
  - [Introduction](#introduction)
  - [MUTTs User Documentation](#mutts-user-documentation)
    - [Prerequisites](#prerequisites)
    - [Installation](#installation)
    - [Updating to the Latest Version](#updating-to-the-latest-version)
    - [Usage](#usage)
      - [Example 1: Generate a JGI Metagenome spreadsheet](#example-1-generate-a-jgi-metagenome-spreadsheet)
      - [Example 2: Generate a JGI Metagenome v15 spreadsheet](#example-2-generate-a-jgi-metagenome-v15-spreadsheet)
      - [Example 3: Generate an EMSL spreadsheet](#example-3-generate-an-emsl-spreadsheet)
      - [Command Options](#command-options)
  - [MUTTs Developer Documentation](#mutts-developer-documentation)
    - [Software Requirements](#software-requirements)
    - [Development Installation](#development-installation)
    - [Running Tests](#running-tests)
    - [Creating Custom Mapper Files](#creating-custom-mapper-files)

## Introduction

The programs bundled in this repository automatically retrieve Biosample metadata records for studies submitted to NMDC through the [NMDC Submission Portal](https://data.microbiomedata.org/submission/home), and convert the metadata into Excel spreadsheets that are accepted by [DOE user facilities](https://www.energy.gov/science/office-science-user-facilities).

---

## MUTTs User Documentation

The documentation and setup instructions in this section are meant for any user who would like to install the MUTTs Python package and use it's transformation capabilities to convert data from the NMDC Submission Portal into an Excel spreadsheet that follows a template, based on the MUTTs JSON mapper file that is used.

### Prerequisites
- [Python](https://www.python.org/downloads/) 3.12 or higher
- An [NMDC user account](https://data.microbiomedata.org/) with an API access token

> To create an NMDC user account you will need to sign up at the above link by clicking on the 'ORCID LOGIN' button/link at the top right corner of the NMDC site, and signing in appropriately with your ORCID credentials

**Setting up your API access token**

This is required for running the examples in the [Usage](#usage) section below (after going through all the [Installation](#installation) steps).

Create a `.env` file in your working directory with the following environment variables:
```bash
echo "DATA_PORTAL_REFRESH_TOKEN=your_token_here" > .env
echo "SUBMISSION_PORTAL_BASE_URL=https://data.microbiomedata.org" >> .env
```

To get your access token:
1. Visit https://data.microbiomedata.org/user
2. Copy your Refresh Token
3. Replace `your_token_here` in the `.env` file with your token

### Installation

1. **Create a virtual environment** (recommended)
```bash
python -m venv mutts-env
source mutts-env/bin/activate  # On Windows: mutts-env\Scripts\activate
```

2. **Install the MUTTs package from PyPI**
```bash
pip install mutts
```

3. **Download any of the MUTTs JSON mapper configuration files**

*Note*: It is not mandatory that you need to download/use any of the pre-existing/already defined JSON mapper files that are present in this repository. You can always define your own custom JSON mapper files that follow a format similar to the ones defined in this repo.

Create a directory for your mapper files and download them from this repository:
```bash
mkdir input-files
cd input-files
```

Download the mapper files you need from the [input-files directory](https://github.com/microbiomedata/metadata-for-user-facility-template-transformations/tree/main/input-files):
- For EMSL: `emsl_header.json`
- For JGI Metagenome: `jgi_mg_header.json` or `jgi_mg_header_v15.json`
- For JGI Metatranscriptome: `jgi_mt_header.json` or `jgi_mt_header_v15.json`

### Updating to the Latest Version

To ensure you have the latest features and bug fixes, you can upgrade the MUTTs package from PyPI:

```bash
pip install --upgrade mutts
```

To check your currently installed version:
```bash
pip show mutts
```

You can also install a specific version if needed:
```bash
pip install mutts==<version>
```

### Usage

Run the `mutts` command with the required options:

```bash
mutts --help
```

Note: In the below examples there is a `--sample-set` argument that requires you to pass it an NMDC Sample Set UUID as value. Each NMDC Submission has one or more Sample Sets associated with it. You can find the Sample Set UUIDs for your Submission by visiting the NMDC Submission Portal and navigating to the Sample Metadata page for a Sample Set within a Submission. The URL will contain the Sample Set UUID.

An example would look like below:

```
https://data.microbiomedata.org/submission/<submission-uuid>/sample_set/<sample-set-uuid>/samples
```

#### Example 1: Generate a JGI Metagenome spreadsheet
```bash
mutts --sample-set <sample-set-uuid> \
      --user-facility jgi_mg \
      --mapper input-files/jgi_mg_header.json \
      --output my-samples_jgi.xlsx
```

#### Example 2: Generate a JGI Metagenome v15 spreadsheet
```bash
mutts --sample-set <sample-set-uuid> \
      --user-facility jgi_mg \
      --mapper input-files/jgi_mg_header_v15.json \
      --output my-samples_jgi_v15.xlsx
```

#### Example 3: Generate an EMSL spreadsheet
```bash
mutts --sample-set <sample-set-uuid> \
      --user-facility emsl \
      --mapper input-files/emsl_header.json \
      --header \
      --output my-samples_emsl.xlsx
```

#### Command Options

- `-s, --sample-set`: Your NMDC sample set UUID (required)
- `-u, --user-facility`: Target facility (required): `emsl`, `jgi_mg`, `jgi_mg_lr`, or `jgi_mt`
- `-m, --mapper`: Path to the JSON mapper file (required)
- `-o, --output`: Output Excel file path (required)
- `-h, --header`: Include headers in output (use for EMSL, omit for JGI)

---

## MUTTs Developer Documentation

The documentation and setup instructions in this section are largely meant for any developer/programmer whose primary use case is to extend/improve/build upon the current capabilities of the MUTTs software.

The software consists of two main components:

1. **JSON Mapper Configuration Files**
- Controls/specifies the mapping between columns from the NMDC Submission Portal and column names used in the output spreadsheets
- Top-level keys indicate main headers in the output
- Numbered keys add clarifying header information
- The `header` keyword allows custom column names
- The `sub_port_mapping` keyword specifies mappings between Submission Portal columns/slots (as dictated by the [NMDC submission schema](https://microbiomedata.github.io/submission-schema/)) and user facility template columns
- Examples available in [input-files/](input-files/)

2. **`mutts` CLI**
- Command-line application that performs the metadata conversion
- Consumes mapper files and submission sample set data as inputs

### Software Requirements
- [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer)
- [Python](https://www.python.org/downloads/release/python-390/) 3.12 or higher

### Development Installation

1. Clone this repository
```bash
git clone https://github.com/microbiomedata/metadata-for-user-facility-template-transformations.git
cd metadata-for-user-facility-template-transformations
```

2. Install dependencies with Poetry
```bash
poetry install
```

This installs the `mutts` package in development mode and creates the `mutts` command-line tool.

3. Set up your `.env` file
```bash
cp .env.example .env  # if available, or create a new .env file
```

Add your NMDC API token and submission portal base URL:
```
DATA_PORTAL_REFRESH_TOKEN=your_token_here
SUBMISSION_PORTAL_BASE_URL=https://data.microbiomedata.org
```

Get your token from: https://data.microbiomedata.org/user

4. Run the CLI in development mode
```bash
poetry run mutts --help
```

### Running Tests

#### Unit Tests

The unit tests are isolated from NMDC services, so they do not require an API token, a `.env` file, or network access. This is also the test suite run for pull requests.

Run the complete unit test suite:

```bash
poetry run pytest
```

Run a single test file:

```bash
poetry run pytest tests/test_dataframe.py
```

Run a single test function:

```bash
poetry run pytest tests/test_dataframe.py::test_merges_environmental_records_by_sample_name
```

#### Integration Test

The integration test makes authenticated, read-only requests to the configured `nmdc-server` instance. It refreshes an access token, reads one stable sample set, and verifies that MUTTs can produce a DataFrame containing an expected sample.

In addition to the URL and token environment variables described above, the integration test requires three additional environment variables. Add these to your `.env` file to run the integration test locally:

```dotenv
INTEGRATION_TEST_SAMPLE_SET_ID=<test-sample-set-id>
INTEGRATION_TEST_USER_FACILITY=<test-user-facility>
INTEGRATION_TEST_EXPECTED_SAMPLE_NAME=<test-sample-name>
```

Then run the integration test with:

```bash
poetry run pytest -m integration
```

Pytest excludes integration tests by default. Selecting the integration marker explicitly overrides that default.

The **Dev nmdc-server integration tests** GitHub Actions workflow runs nightly and can also be started manually. It is configured through repository secrets and variables to talk to the deployed dev `nmdc-server` instance. The workflow tests both the default-branch (`main`) source and the latest published MUTTs package:

| Published MUTTs | Default-branch MUTTs | Meaning                                                                                               |
|-----------------|----------------------|-------------------------------------------------------------------------------------------------------|
| Pass            | Pass                 | Dev `nmdc-server` remains compatible with current and future MUTTs.                                   |
| Fail            | Pass                 | A compatible MUTTs change exists but has not been published.                                          |
| Fail            | Fail                 | The `nmdc-server` contract, deployment, authentication, fixture, or shared client path may be broken. |
| Pass            | Fail                 | Published users remain safe; MUTTs development has regressed.                                         |

Failures create or update one GitHub issue. A successful recovery comments on and closes that issue.

### Creating Custom Mapper Files

To create a custom mapper for a new user facility, refer to the existing examples:
- [emsl_header.json](input-files/emsl_header.json) - EMSL configuration
- [jgi_mg_header.json](input-files/jgi_mg_header.json) - JGI Metagenome configuration
- [jgi_mt_header.json](input-files/jgi_mt_header.json) - JGI Metatranscriptome configuration
- [jgi_mg_header_v15.json](input-files/jgi_mg_header_v15.json) - JGI Metagenome v15 configuration
- [jgi_mt_header_v15.json](input-files/jgi_mt_header_v15.json) - JGI Metatranscriptome v15 configuration

