Metadata-Version: 2.4
Name: marinerg-data
Version: 0.3.0
Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
License: MIT
Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jsonschema>=4
Requires-Dist: requests
Requires-Dist: PyYAML
Requires-Dist: pydantic>=2
Requires-Dist: marinerg-schema>=0.2.0
Provides-Extra: mock
Requires-Dist: h5netcdf>=1.3; extra == "mock"
Requires-Dist: numpy; extra == "mock"
Provides-Extra: kerchunk
Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
Requires-Dist: h5py>=3.10; extra == "kerchunk"
Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
Requires-Dist: zarr>=3; extra == "kerchunk"
Requires-Dist: xarray>=2024.1; extra == "kerchunk"
Requires-Dist: h5netcdf>=1.3; extra == "kerchunk"
Requires-Dist: numpy>=2.4.6; extra == "kerchunk"
Provides-Extra: vre
Requires-Dist: kerchunk>=0.2.6; extra == "vre"
Requires-Dist: h5py>=3.10; extra == "vre"
Requires-Dist: fsspec>=2024.2; extra == "vre"
Requires-Dist: zarr>=3; extra == "vre"
Requires-Dist: xarray>=2024.1; extra == "vre"
Requires-Dist: h5netcdf>=1.3; extra == "vre"
Requires-Dist: intake>=2; extra == "vre"
Requires-Dist: intake-xarray; extra == "vre"
Requires-Dist: matplotlib>=3.8; extra == "vre"
Requires-Dist: jinja2>=3; extra == "vre"
Requires-Dist: aiohttp>=3.9; extra == "vre"
Requires-Dist: numpy>=2.4.6; extra == "vre"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Requires-Dist: types-PyYAML; extra == "types"
Provides-Extra: tabular
Requires-Dist: openpyxl>=3.1; extra == "tabular"
Requires-Dist: scipy>=1.11; extra == "tabular"
Dynamic: license-file

# marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: **inspect →
convert → validate → publish**, plus VRE consumption (search, resolve, fetch,
lazy streaming). Django-free and pip-installable, so it runs both at a test
facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: **bulk data never transits the
e-infrastructure.** Facilities publish to their own Zenodo; this tool registers
the record in CKAN; the server only re-validates metadata.

## Install

```sh
pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install "marinerg-data[vre]"          # + complete Blue-Cloud / Jupyter runtime
pip install marinerg-wave-tank            # optional wave-tank converter family
```

The lightweight core client supports Python 3.11 or newer and has no
notebook, NetCDF, or container dependency. `kerchunk`, `mock`, and `vre` are
explicit opt-ins. Windows is a supported installation target for the core
client; the release gate includes a Windows install-and-validate smoke test.

## Commands

| Command | Purpose |
|---|---|
| `inspect <dir>` | Sniff capture formats, report converter families; `--draft-manifest` writes a skeleton |
| `convert <run-dir>` | Run a converter family (`--family wave-tank`, else auto-detect); writes open-format outputs + `<run>.manifest.json`, then validates. With `[kerchunk]`, also emits a `<file>.nc.kerchunk.json` reference sidecar per NetCDF |
| `validate <manifest-or-dir>` | JSON Schema (run-manifest v0.1) + referential integrity + file existence, and `dataset.yaml`'s controlled fields against the catalogue's vocabularies; `--verify-checksums`, `--no-files` |
| `publish <bundle-dir>` | Facility Zenodo record + CKAN registration; `--sandbox`, `--new-version`, `--dry-run` |
| `fetch <doi> --cache <dir>` | Download a published dataset into a content-addressed shared cache (keyed by DOI + optional manifest checksum); first lab member pays the download, the rest hit the local copy. `--from-ckan <name> --ckan-url <url>` resolves the DOI via CKAN; `--sandbox` |
| `catalog --ckan-url <url>` | Generate an [intake](https://intake.readthedocs.io) catalogue from CKAN so MARINERG-i datasets open natively in a VRE (`intake.open_catalog(...)` → `.to_dask()`); prefers kerchunk sidecars (lazy `zarr`) over direct NetCDF |
| `mock --scenario <name>` | Synthetic NetCDF + manifest for fixtures/seeding/demos (`--scenario list`) |
| `template list` | Every equipment, facility and campaign template, and whether their references resolve |
| `template check <id> <file>` | Read a real export through a template and report what it mapped, what it could not, and what a reader would be left guessing — the loop for onboarding a facility |
| `report <campaign> --out <dir>` | Generate a campaign's raw handover: a folder per instrument in each vendor's own shape, plus the operator's log |
| `standardise <campaign> <raw-dir> --out <dir>` | Read a handover through its templates and write NetCDF-4, a kerchunk sidecar and one run manifest per run, with each source's declared clock offset applied; also writes `dataset.yaml`, so the output is a publishable bundle |
| `capture --scenario <name>` | A single messy export (CSV/XLSX/MATLAB) to run `convert` against, including one that fails validation |

Environment: `ZENODO_TOKEN` (a sandbox token with `--sandbox`), `CKAN_URL`,
`CKAN_TOKEN`.

## Onboarding a facility

Equipment templates describe how a vendor writes its export — header style,
delimiter, the metadata keys it invents, what its columns mean — so a new
format is a YAML file rather than a converter. Facility templates hold what a
facility restates on every campaign: identity, ROR, site, default rights.
Campaign templates say which instruments a test report spans.

```sh
marinerg-data template list
marinerg-data template check qualisys-tsv ./run01.tsv
```

Sample raw handovers are in [`demo/raw/`](demo/raw/) — small enough to read,
and realistic: every instrument a different shape, the sea state only in the
operator's spreadsheet, and the clocks disagreeing.

## Publishing is resumable

A Zenodo record cannot be withdrawn, so `publish` records each step before the
one it cannot undo. The deposition goes into `<bundle>/.marinerg-publish-pending.json`
before it is published, and the minted DOI into `<bundle>/.marinerg-publish.json`
before CKAN is called. **Rerun `publish` on the same bundle after any failure** —
it resumes from wherever it stopped and never mints a second record for the same
data, including when a publish request times out after Zenodo has committed.
Both files are local bookkeeping and are never uploaded.

## Cloud-optimised access (VRE)

With the `[kerchunk]` extra, `convert` emits a kerchunk reference sidecar per
NetCDF. The sidecar lets `xarray` lazily stream individual chunks straight from
Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one
gauge, transfer megabytes not gigabytes. The reference URL is templated (`{{u}}`)
so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

## Examples

Runnable VRE notebooks in [`examples/`](examples/): lazy kerchunk streaming and
intake-based discovery, each self-contained against synthetic data. See
[`examples/README.md`](examples/README.md).

## Blue-Cloud / D4Science handover

The operator-facing install profile, CKAN/DCAT endpoint contract, and live-demo
release checklist are in [`docs/blue-cloud-handover.md`](docs/blue-cloud-handover.md).
The VRE needs Python 3.11 or newer, outbound HTTPS to the MARINERG-i catalogue,
Zenodo, and PyPI, and persistent shared workspace storage.  No MARINERG-i data
hosting, inbound connectivity, or identity federation is required for the
public-data workflow.

## Develop

```sh
uv sync --group dev
uv run pytest          # Django-free, no internet (HTTP is faked; the Zenodo
                       # contract test spawns a local zenodo-mock)
uv run ruff check src tests
```

Development and CI use Python 3.14 with the uv-native toolchain; the published
client supports Python 3.11 or newer. See `CLAUDE.md` for architecture pointers
and the token-free release flow.

## Licence

Copyright ICHEC. MIT — see [`LICENSE`](LICENSE).
