Metadata-Version: 2.4
Name: datamanifestpy
Version: 0.18.1
Summary: Python port of DataManifest.jl — declare and manage data dependencies for scientific projects
Author-email: Mahé Perrette <mahe.perrette@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Mahé Perrette
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/perrette/datamanifest
Project-URL: Repository, https://github.com/perrette/datamanifest
Project-URL: Issues, https://github.com/perrette/datamanifest/issues
Project-URL: Documentation, https://perrette.github.io/datamanifest/
Project-URL: Changelog, https://github.com/perrette/datamanifest/commits/main
Keywords: data,dataset,data-management,data-dependencies,manifest,datamanifest,reproducibility,reproducible-research,provenance,checksum,doi,download,fair-data,scientific-computing,research-data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Archiving :: Mirroring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx
Requires-Dist: tqdm
Requires-Dist: tomli_w
Requires-Dist: tomli; python_version < "3.11"
Requires-Dist: platformdirs
Provides-Extra: csv
Requires-Dist: pandas; extra == "csv"
Provides-Extra: parquet
Requires-Dist: pandas; extra == "parquet"
Requires-Dist: pyarrow; extra == "parquet"
Provides-Extra: nc
Requires-Dist: xarray; extra == "nc"
Requires-Dist: netcdf4; extra == "nc"
Provides-Extra: yaml
Requires-Dist: pyyaml; extra == "yaml"
Provides-Extra: fsspec
Requires-Dist: fsspec; extra == "fsspec"
Provides-Extra: all
Requires-Dist: pandas; extra == "all"
Requires-Dist: pyarrow; extra == "all"
Requires-Dist: xarray; extra == "all"
Requires-Dist: netcdf4; extra == "all"
Requires-Dist: pyyaml; extra == "all"
Requires-Dist: fsspec; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: fsspec; extra == "dev"
Requires-Dist: pyyaml; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.5; extra == "docs"
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocs-include-markdown-plugin>=6.0; extra == "docs"
Dynamic: license-file

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/perrette/datamanifest.toml/main/design/logo/lockup-dark.svg">
    <img src="https://raw.githubusercontent.com/perrette/datamanifest.toml/main/design/logo/lockup.svg" alt="datamanifest.toml" height="76">
  </picture>
</p>

# datamanifest[py]

[![pypi](https://img.shields.io/pypi/v/datamanifestpy)](https://pypi.org/project/datamanifestpy)
![python](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fperrette%2Fdatamanifest%2Frefs%2Fheads%2Fmain%2Fpyproject.toml)
[![CI](https://github.com/perrette/datamanifest/actions/workflows/ci.yaml/badge.svg)](https://github.com/perrette/datamanifest/actions/workflows/ci.yaml)
[![docs](https://img.shields.io/badge/docs-perrette.github.io%2Fdatamanifest-blue)](https://perrette.github.io/datamanifest/)

Keep track of the datasets used in a scientific project. You declare your data
dependencies — URLs, git repositories, checksums, formats — in a
`datamanifest.toml` file; `datamanifest` downloads, verifies, extracts and loads
them, and caches your own computed results with the same machinery.

<!-- intro-start -->
- **A transparent, trackable manifest.** Every dataset a project depends on —
  URLs, DOIs, checksums (content hashes used to verify each file), formats — is
  listed in a single `datamanifest.toml` file (the *manifest*) that you can read
  directly and version with git. The format is defined by a
  [language-agnostic spec](https://github.com/perrette/datamanifest.toml/blob/main/SCHEMA.md)
  (implemented in Python and Julia) and can be edited by hand, from code, or
  through the CLI.
- **Fetch from a wide range of sources.** Direct URLs, Zenodo/figshare and PANGAEA
  DOIs, git repos, object stores (`s3://`, `gs://`, …), and bulk imports from pooch, intake
  or DVC — all checksum-verified, extracted, and adopted in place when already on
  disk.
- **Cache your own computed data too.** The same tooling backs a `@cached`
  decorator that stores your own results with PID-lock, keyed by their inputs, to speed up
  calculations locally. Caching is a separate, local concern — nothing is
  fetched — but cached results are managed through the same CLI as datasets.
- **A CLI for data download, local management and synchronization across
  machines.** Add and download datasets, inspect and repair what's on disk, move
  or centralize where data is stored, and push/pull datasets and cached results
  between machines over rsync+ssh — all without touching your analysis code. A
  git-ignored *state file* (`.datamanifest/state.toml`) records where each object
  actually landed on this machine, keeping local location tracking separate from
  the portable, shareable manifest.
<!-- intro-end -->

## Installation

```bash
pip install datamanifestpy
```

With optional loader backends (`csv`, `parquet`, `nc`, `yaml`, `fsspec`, or
`all`):

```bash
pip install "datamanifestpy[all]"
```

See the [installation page](https://perrette.github.io/datamanifest/installation/)
for the per-backend details.

## Quickstart

```bash
datamanifest init                  # create datamanifest.toml here
datamanifest add https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv --name co2
datamanifest list                  # what's tracked, and where it lives
datamanifest path co2              # resolve the on-disk path (for a script)
```

Then load it from your code:

```python
import datamanifest

db = datamanifest.Database("datamanifest.toml")
df = db.load_dataset("co2")          # download on first use, then load
path = db.get_dataset_path("co2")    # just the on-disk path
```

**Commit `datamanifest.toml`** — the recipe of what to fetch and how. The data
lives in a machine-wide shared store (deduplicated across your projects) and
the private `.datamanifest/` directory stays git-ignored; a collaborator clones
and runs `datamanifest download`. See the
[quickstart](https://perrette.github.io/datamanifest/quickstart/) for the full
walkthrough.

## Documentation

Full documentation lives at **<https://perrette.github.io/datamanifest/>**:

- [Installation](https://perrette.github.io/datamanifest/installation/)
- [Quickstart](https://perrette.github.io/datamanifest/quickstart/)
- [Using it from your code](https://perrette.github.io/datamanifest/api/) — `load_dataset`, `@cached`, the file-less `Database`
- [Use cases](https://perrette.github.io/datamanifest/use-cases/) — add, repair, store, sync
- [CLI reference](https://perrette.github.io/datamanifest/cli/)
- [Storage model](https://perrette.github.io/datamanifest/storage/) · [Configuration](https://perrette.github.io/datamanifest/configuration/)
- [Adding datasets](https://perrette.github.io/datamanifest/adding-datasets/) · [Importing from other tools](https://perrette.github.io/datamanifest/importing/)
- [Language bindings](https://perrette.github.io/datamanifest/language-bindings/) · [Related projects](https://perrette.github.io/datamanifest/related/)

## From the same author

A few other open-source tools I maintain.

**Scientific writing & data**

- [**texmark**](https://perrette.github.io/texmark/) — write scientific articles in Markdown and convert them to journal-ready LaTeX/PDF.
- [**papers**](https://perrette.github.io/papers/) — command-line BibTeX bibliography and PDF library manager.

**Speech to Text (dictate) and Text to Speech (read-aloud) tools**

- [**scribe**](https://perrette.github.io/scribe/) — speech-to-text dictation.
- [**bard**](https://perrette.github.io/bard/) — text-to-speech reader.

## Acknowledgments

`datamanifest` is a Python port of
[`awi-esc/DataManifest.jl`](https://github.com/awi-esc/DataManifest.jl), written
by the same author (Mahé Perrette). The Python port was implemented with
assistance from [Anthropic's Claude](https://www.anthropic.com/claude).
