Metadata-Version: 2.1
Name: quilt3distribute
Version: 0.1.2
Summary: A small wrapper around quilt3 to make distributing manifest style datasets even easier.
Home-page: https://github.com/AllenCellModeling/quilt3distribute
Author: Jackson Maxfield Brown
Author-email: jacksonb@alleninstitute.org
License: Allen Institute Software License
Keywords: quilt3distribute
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: Free for non-commercial use
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: markdown2 (>=2.3.7)
Requires-Dist: pandas
Requires-Dist: python-dateutil (==2.8.0)
Requires-Dist: quilt3 (>=3.0.4)
Requires-Dist: tabulate (>=0.8.3)
Requires-Dist: tqdm (>=4.32.1)
Provides-Extra: all
Requires-Dist: markdown2 (>=2.3.7) ; extra == 'all'
Requires-Dist: pandas ; extra == 'all'
Requires-Dist: python-dateutil (==2.8.0) ; extra == 'all'
Requires-Dist: quilt3 (>=3.0.4) ; extra == 'all'
Requires-Dist: tabulate (>=0.8.3) ; extra == 'all'
Requires-Dist: tqdm (>=4.32.1) ; extra == 'all'
Requires-Dist: codecov ; extra == 'all'
Requires-Dist: flake8 ; extra == 'all'
Requires-Dist: numpy ; extra == 'all'
Requires-Dist: pytest ; extra == 'all'
Requires-Dist: pytest-cov ; extra == 'all'
Requires-Dist: pytest-raises ; extra == 'all'
Requires-Dist: tifffile (==0.15.1) ; extra == 'all'
Requires-Dist: pytest-runner ; extra == 'all'
Requires-Dist: bumpversion (>=0.5.3) ; extra == 'all'
Requires-Dist: coverage (>=5.0a4) ; extra == 'all'
Requires-Dist: flake8 (>=3.7.7) ; extra == 'all'
Requires-Dist: ipython (>=7.5.0) ; extra == 'all'
Requires-Dist: m2r (>=0.2.1) ; extra == 'all'
Requires-Dist: pytest (>=4.3.0) ; extra == 'all'
Requires-Dist: pytest-cov (==2.6.1) ; extra == 'all'
Requires-Dist: pytest-raises (>=0.10) ; extra == 'all'
Requires-Dist: pytest-runner (>=4.4) ; extra == 'all'
Requires-Dist: Sphinx (>=2.0.0b1) ; extra == 'all'
Requires-Dist: sphinx-rtd-theme (>=0.1.2) ; extra == 'all'
Requires-Dist: tox (>=3.5.2) ; extra == 'all'
Requires-Dist: twine (>=1.13.0) ; extra == 'all'
Requires-Dist: wheel (>=0.33.1) ; extra == 'all'
Requires-Dist: altair ; extra == 'all'
Requires-Dist: jupyterlab ; extra == 'all'
Requires-Dist: matplotlib ; extra == 'all'
Provides-Extra: dev
Requires-Dist: bumpversion (>=0.5.3) ; extra == 'dev'
Requires-Dist: coverage (>=5.0a4) ; extra == 'dev'
Requires-Dist: flake8 (>=3.7.7) ; extra == 'dev'
Requires-Dist: ipython (>=7.5.0) ; extra == 'dev'
Requires-Dist: m2r (>=0.2.1) ; extra == 'dev'
Requires-Dist: pytest (>=4.3.0) ; extra == 'dev'
Requires-Dist: pytest-cov (==2.6.1) ; extra == 'dev'
Requires-Dist: pytest-raises (>=0.10) ; extra == 'dev'
Requires-Dist: pytest-runner (>=4.4) ; extra == 'dev'
Requires-Dist: Sphinx (>=2.0.0b1) ; extra == 'dev'
Requires-Dist: sphinx-rtd-theme (>=0.1.2) ; extra == 'dev'
Requires-Dist: tox (>=3.5.2) ; extra == 'dev'
Requires-Dist: twine (>=1.13.0) ; extra == 'dev'
Requires-Dist: wheel (>=0.33.1) ; extra == 'dev'
Provides-Extra: interactive
Requires-Dist: altair ; extra == 'interactive'
Requires-Dist: jupyterlab ; extra == 'interactive'
Requires-Dist: matplotlib ; extra == 'interactive'
Provides-Extra: setup
Requires-Dist: pytest-runner ; extra == 'setup'
Provides-Extra: test
Requires-Dist: codecov ; extra == 'test'
Requires-Dist: flake8 ; extra == 'test'
Requires-Dist: numpy ; extra == 'test'
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: pytest-cov ; extra == 'test'
Requires-Dist: pytest-raises ; extra == 'test'
Requires-Dist: tifffile (==0.15.1) ; extra == 'test'

# quilt3distribute

[![Build Status](https://github.com/AllenCellModeling/quilt3distribute/workflows/Build/badge.svg)](https://github.com/AllenCellModeling/quilt3distribute/actions)
[![Documentation](https://github.com/AllenCellModeling/quilt3distribute/workflows/Documentation/badge.svg)](https://AllenCellModeling.github.io/quilt3distribute)
[![Code Coverage](https://codecov.io/gh/AllenCellModeling/quilt3distribute/branch/master/graph/badge.svg)](https://codecov.io/gh/AllenCellModeling/quilt3distribute)
<br>
![dataset packaging and distribution](http://www.allencell.org/uploads/8/1/9/9/81996008/published/automatingaccess-button-3_2.png?1549322257)

People commonly work with tabular datasets, people want to share their data, this makes that easier through Quilt3.

---

## Features
* Automatically determines which files to upload based off CSV headers. (Explicit override available)
* Simple interface for attaching metadata to each file based off the manifest contents.
* Groups metadata for files that are referenced multiple times.
* Validates and runs basic cleaning operations on your dataset manifest CSV.
* Optionally add license details and usage instructions to your dataset README.
* Parses README for any referenced files and packages them up as well.
* Support for adding extra files not contained in the manifest.
* Constructs an "associates" map that is placed into each files metadata for quick navigation around the package.
* Enforces that the metadata attached to each file is standardized across the package for each file column.

## Quick Start
Construct a csv (or pandas dataframe) dataset manifest ([Example](quilt3distribute/tests/data/example.csv)):

| CellId | Structure | 2dReadPath | 3dReadPath |
|--------|-----------|------------|------------|
| 1      | lysosome  | 2d/1.png   | 3d/1.tiff  |
| 2      | laminb1   | 2d/2.png   | 3d/2.tiff  |
| 3      | golgi     | 2d/3.png   | 3d/3.tiff  |
| 4      | myosin    | 2d/4.png   | 3d/4.tiff  |

```python
from quilt3distribute import Dataset

# Create the dataset
ds = Dataset(
    dataset="single_cell_examples.csv",
    name="single_cell_examples",
    package_owner="jacksonb",
    readme_path="single_cell_examples.md"
)

# Optionally add common additional requirements
ds.add_usage_doc("https://docs.quiltdata.com/walkthrough/reading-from-a-package")
ds.add_license("https://www.allencell.org/terms-of-use.html")

# Optionally indicate column values to use for file metadata
ds.set_metadata_columns(["CellId", "Structure"])

# Optionally rename the columns on the package level
ds.set_column_names_map({
    "2dReadPath": "images_2d",
    "3dReadPath": "images_3d"
})

# Distribute
pkg = ds.distribute(push_uri="s3://quilt-jacksonb", message="Initial dataset example")
```

***Returns:***
```
(remote Package)
 └─README.md
 └─images_2d
   └─03cdf019_1.png
   └─148ddc09_2.png
   └─2b2cf361_3.png
   └─312a0367_4.png
 └─images_3d
   └─a0ce6e01_1.tiff
   └─c360072c_2.tiff
   └─d9b55cba_3.tiff
   └─eb29e6b3_4.tiff
 └─metadata.csv
 └─referenced_files
   └─some_file_referenced_by_the_readme.png
```

***Example Metadata:***
```python
pkg["images_2d"]["03cdf019_1.png"].meta
```
```json
{
    "CellId": 1,
    "Structure": "lysosome",
    "associates": {
        "images_2d": "images_2d/03cdf019_1.png",
        "images_3d": "images_3d/a0ce6e01_1.tiff"
    }
}
```

## Installation
**Stable Release:** `pip install quilt3distribute`<br>
**Development Head:** `pip install git+https://github.com/AllenCellModeling/quilt3distribute.git`


### Credits

This package was created with Cookiecutter. [Original repository](https://github.com/audreyr/cookiecutter)


***Free software: Allen Institute Software License***


