Metadata-Version: 2.4
Name: sciharvest
Version: 0.1.0
Summary: Zero-key, reproducible harvesting of public scientific datasets with auditable provenance.
Author: Yanxin Zhao
License-Expression: MIT
Project-URL: Homepage, https://github.com/kyoai-zhao/sciharvest
Keywords: reproducibility,open-data,provenance,research-data,scientific-computing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# sciharvest

**Zero-key, reproducible harvesting of public scientific datasets with auditable provenance.**

`sciharvest` turns a tiny JSON "descriptor" (which source, which query) into a
portable, self-describing dataset: every harvest writes a normalized payload
plus a provenance **manifest** that records the request, the fetch time, the
tool version, the source license, and a SHA-256 of the stored data. Re-run
`verify` and `reproduce` at any time to prove the data hasn't drifted.

It has **zero runtime dependencies** (Python standard library only), which
makes it trivial to install, audit, and run in air-gapped or long-term
research environments.

[![CI](https://github.com/kyoai-zhao/sciharvest/actions/workflows/ci.yml/badge.svg)](https://github.com/kyoai-zhao/sciharvest/actions/workflows/ci.yml)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

---

## Why

Reproducible research starts at the data boundary. All too often a figure's
data lives in a file with no record of *where it came from, when it was pulled,
or under what license it was reused*. `sciharvest` encodes that boundary:

- **Re-run nobody's logs** — a descriptor fully names the query.
- **Prove provenance** — the manifest hash means verifiable, self-describing data.
- **Detect drift** — upstream data changes are caught by `reproduce`, not discovered in review.

## Features

- No API keys needed for any built-in source (USGS, OpenAlex, PubMed, Crossref).
- Zero runtime dependencies: pure `urllib` + `unittest`.
- Deterministic payload hashing (SHA-256) for offline re-verification.
- `fetch` / `verify` / `reproduce` lifecycle that treats data as code.
- Polite-pool etiquette: identifies itself (User-Agent, NCBI `tool`/`email`,
  Crossref `mailto`) and rate-limits NCBI calls.
- Recorded offline fixtures so the test suite never needs the network.

## Install & Quickstart

```console
$ git clone https://github.com/kyoai-zhao/sciharvest.git
$ cd sciharvest
$ python -m pip install -e .
$ sciharvest --version
sciharvest 0.1.0
```

Fetch an example dataset (USGS daily discharge, Potomac River):

```console
$ sciharvest fetch examples/usgs-potomac-discharge.json --out ./data
potomac-daily-discharge-60d: 59 rows from usgs
payload_sha256: f3f615a02bb978ba28cef96d208e4658d33a5db1d35bcc73557aa4896cf54244
manifest:       ./data/manifests/potomac-daily-discharge-60d.json
```

Verify that stored data still matches its manifest:

```console
$ sciharvest verify examples/usgs-potomac-discharge.json --out ./data
potomac-daily-discharge-60d: OK: stored data matches the manifest (59 rows)
```

Re-fetch and detect upstream drift:

```console
$ sciharvest reproduce examples/usgs-potomac-discharge.json --out ./data
```

The dataset is now `./data`:

```
data/
├── raw/
│   └── potomac-daily-discharge-60d.json     # normalized rows + columns
└── manifests/
    └── potomac-daily-discharge-60d.json     # provenance manifest (hash, request, license, time)
```

## CLI reference

| Command | Purpose |
| --- | --- |
| `sciharvest fetch <descriptor> [--out DIR] [--preview N]` | Harvest and store a dataset. |
| `sciharvest verify <descriptor> [--out DIR]` | Hash stored data against its manifest (`ok`/`missing`/`drifted`; non-zero exit on drift). |
| `sciharvest reproduce <descriptor> [--out DIR]` | Re-fetch and report drift vs. the previous run. |
| `sciharvest list-sources` | List registered data sources. |
| `sciharvest describe <source>` | Show one source's schema, license, and docs. |

## Dataset descriptors

A descriptor is the reproducible recipe — a small JSON object:

```json
{
  "name": "potomac-daily-discharge-60d",
  "source": "usgs",
  "request": {
    "endpoint": "dv",
    "params": { "sites": "01646500", "parameterCd": "00060", "period": "P60D" }
  },
  "notes": "Daily-mean discharge at Potomac River near Washington, DC.",
  "license": "Public domain (U.S. Government work)"
}
```

| Field | Required | Meaning |
| --- | --- | --- |
| `name` | yes | Slug used as the file name (letters, digits, `.`, `_`, `-`). |
| `source` | yes | A registered source (see `sciharvest list-sources`). |
| `request` | yes | Source-specific query; validated up front. |
| `notes` | no | Free-text provenance annotation. |
| `license` | no | Overrides the source's default license in the manifest. |

## Provenance manifest

Every payload is paired with an auditable manifest:

```json
{
  "schema": "sciharvest/manifest.v1",
  "name": "potomac-daily-discharge-60d",
  "source": "usgs",
  "request": { "endpoint": "dv", "params": { "period": "P60D", ... } },
  "fetched_at": "2026-08-13T20:10:00Z",
  "tool_version": "0.1.0",
  "license": "Public domain (U.S. Government work)",
  "source_homepage": "https://waterdata.usgs.gov/",
  "source_documentation": "https://waterservices.usgs.gov/rest/",
  "record_count": 59,
  "columns": ["site_no", "variable", "unit", "datetime", "value", "qualifier"],
  "unit": "ft3/s",
  "hash_algorithm": "sha256",
  "payload_sha256": "f3f615a0..."
}
```

`unit` is read from the source payload (never assumed) and echoed into every
row — a defensive check against silent unit mismatches in later analysis.

## Built-in sources

| Source | What it provides | License | Keys? |
| --- | --- | --- | --- |
| `usgs` | USGS NWIS: instantaneous (`iv`), daily (`dv`), site metadata | Public domain | no |
| `openalex` | Scholarly works search (title, year, citations, authors, DOI) | CC0 1.0 | no |
| `pubmed` | PubMed article metadata via NCBI E-utilities | NCBI policies | no |
| `crossref` | Authoritative metadata for one DOI | Crossref REST | no |

## Development

```console
$ python -m unittest discover -s tests -t .       # offline fixture-based suite
$ python -m pip install -e .
$ sciharvest list-sources
```

The suite never touches the network; fixtures in `tests/fixtures/` are pruned,
real API responses. To add a source, see [`CONTRIBUTING.md`](CONTRIBUTING.md).

## Roadmap

- High-water-mark de-duplication and `dataset` diff output
- CSV/Parquet exporters with a stable column contract
- Descriptor validation schema convention (`$schema`)
- More keyless sources (e.g., US Census, Open-Meteo, UNdata)

## License

MIT — see [`LICENSE`](LICENSE). Upstream data retains its own license, which is
recorded in every manifest.
