Metadata-Version: 2.4
Name: ncarnate
Version: 2.2.2
Summary: Convert HDF4/HDF-EOS2 (MODIS, AMSR-E) to CF-annotated netCDF4 and losslessly recompress netCDF/HDF5 files.
Project-URL: Homepage, https://github.com/ErickShepherd/ncarnate
Project-URL: Documentation, https://ncarnate.readthedocs.io/en/stable/
Project-URL: Source, https://github.com/ErickShepherd/ncarnate
Project-URL: Changelog, https://github.com/ErickShepherd/ncarnate/releases
Project-URL: Issues, https://github.com/ErickShepherd/ncarnate/issues
Author-email: Erick Shepherd <dev@erickshepherd.com>
License-Expression: MIT
License-File: LICENSE
Keywords: AMSR-E,CF conventions,GIS,HDF-EOS,HDF-EOS2,HDF4,HDF5,MODIS,atmospheric science,compression,geolocation,grid,netCDF,netCDF4,recompression,remote sensing,swath,xarray
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.10
Requires-Dist: netcdf4>=1.6
Requires-Dist: numpy>=1.26
Requires-Dist: pyhdf>=0.11.6
Requires-Dist: pyproj>=3.6
Requires-Dist: tqdm>=4.66
Provides-Extra: test
Requires-Dist: build>=1.0; extra == 'test'
Requires-Dist: hatchling>=1.27; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ErickShepherd/ncarnate/main/brand/ncarnate-lockup-dark.png">
    <img alt="ncarnate" src="https://raw.githubusercontent.com/ErickShepherd/ncarnate/main/brand/ncarnate-lockup.png" width="460">
  </picture>
</p>

[![CI status](https://github.com/ErickShepherd/ncarnate/actions/workflows/ci.yml/badge.svg)](https://github.com/ErickShepherd/ncarnate/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/ncarnate.svg)](https://pypi.org/project/ncarnate/)
[![conda-forge version](https://img.shields.io/conda/vn/conda-forge/ncarnate.svg)](https://anaconda.org/conda-forge/ncarnate)
[![Docs](https://readthedocs.org/projects/ncarnate/badge/?version=latest)](https://ncarnate.readthedocs.io/en/latest/)
[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/ErickShepherd/ncarnate/blob/main/LICENSE)
[![Python 3.10–3.13](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)](https://pypi.org/project/ncarnate/)
[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.21288802-blue)](https://doi.org/10.5281/zenodo.21288802)

Reincarnate legacy scientific data as modern netCDF4.

ncarnate reads netCDF3, netCDF4/HDF5, and HDF4/HDF-EOS2 files and writes
recompressed, CF-annotated netCDF4. It does two jobs:

- **Recompress** netCDF/HDF5 files — change the compression level, shuffle
  filter, or storage layout without changing a single stored value.
- **Convert** HDF4 and HDF-EOS2 granules (AMSR-E, MODIS, and kin) to netCDF4,
  reconstructing the CF coordinates that modern tools (xarray, QGIS, Panoply)
  need: grid projections become CF grid mappings with 1-D `x`/`y` and 2-D
  `lat`/`lon` coordinates, swath geolocation is attached as CF coordinates, and
  dimension-mapped (e.g. 5 km → 1 km) geolocation is interpolated through ECEF
  space.

## Problems this solves

Reach for ncarnate if you are trying to:

- **Convert HDF4 / HDF-EOS2 granules (MODIS, AMSR-E, and kin) to netCDF4** so
  they open cleanly in xarray, QGIS, or Panoply.
- **Read an HDF-EOS2 swath or grid that has no usable lat/lon** — ncarnate
  reconstructs CF `lat`/`lon` coordinates and grid mappings so the data is
  actually georeferenced, instead of an unplottable array.
- **Recompress a netCDF4 / HDF5 file** — change the compression level or shuffle
  filter without altering a single stored value.
- **Shrink an archive of scientific files** without risking the science: every
  output is verified value-for-value against its source before it replaces
  anything, and stored values round-trip value-identically (bit-for-bit for
  integer and packed data; NaN- and signed-zero-insensitive for floating point).
- **Batch-convert a directory tree** of legacy granules to modern netCDF4 in one
  command.

## The fidelity contract

Converting or recompressing a file changes *storage*, never *science data*:

- Every variable's stored values are preserved **value-identically** — bit-for-bit
  for integer and packed data; for floating-point data, distinct NaN
  bit-patterns and `-0.0`/`+0.0` compare equal. Packed integers stay packed;
  `scale_factor`/`add_offset`/`_FillValue` are carried across as declarations,
  never applied.
- Every dimension (including unlimited-ness), attribute (including its exact
  storage type — an `NC_STRING` scalar stays `NC_STRING`, verified via netCDF-C
  type inquiry), and group survives. HDF-EOS2 `StructMetadata` is preserved
  verbatim; names netCDF cannot hold are sanitized with the original recorded in
  a companion attribute.
- **Complex-valued variables (`complex64`/`complex128`) are excluded** from the
  fidelity guarantee: netCDF stores them as compound types, which ncarnate
  **refuses loudly** with the stable `UNSUPPORTED_TYPE` error rather than
  guessing at a lossy copy. Complex support is a later, evidence-backed feature.
- Geolocation reconstruction is strictly **additive**: the original information
  always rides along, so the conversion never becomes the only copy of the
  truth. Swath coordinates are attached to variables whose first two axes are
  the swath axes; a variable with a leading band/byte dimension is converted
  intact but gets no `coordinates` attribute (a warning says so).
- Every output is **verified against the source value-for-value before it
  replaces anything**. A source file is never destroyed by a failed run, and
  HDF4 sources are never replaced at all.
- Unsupported constructs (user-defined netCDF types, unverified GCTP projections,
  exotic swath layouts) **fail loud** with a named error rather than guessing — a
  wrong coordinate is worse than a refused conversion. `--no-geolocation`
  converts the raw payload anyway.

The details, the guarantee boundary, and how the test suite pins each clause live
in [`docs/fidelity-notes.md`](https://github.com/ErickShepherd/ncarnate/blob/main/docs/fidelity-notes.md).

## Installation

**With conda** (from [conda-forge](https://anaconda.org/conda-forge/ncarnate)):

```console
conda install -c conda-forge ncarnate
```

This works on every platform and is the recommended install on **Windows** —
conda-forge's `pyhdf` is built against a proper HDF4 library everywhere, so the
full HDF4/HDF-EOS2 converter runs on Windows, macOS, and Linux alike.

**With pip** (from [PyPI](https://pypi.org/project/ncarnate/)):

```console
pip install ncarnate
```

On **Linux (x86_64)** and **macOS (arm64)**, every dependency — including
`pyhdf` — installs as a self-contained binary wheel with no system libraries
required. On platforms without a repaired `pyhdf` wheel (e.g. Linux aarch64),
building from sdist requires the system HDF4 library first (Debian/Ubuntu:
`apt install libhdf4-dev`).

**Windows via pip:** PyPI wheels give you the full netCDF/HDF5 surface —
`import ncarnate`, the CLI (`--help`/`--version`), format detection, audits,
manifest runs, and verified recompression — but **not** HDF4/HDF-EOS2
*conversion*: `pyhdf`'s Windows wheel ships no HDF4 runtime. An HDF4 attempt
is refused cleanly **before any output is created** with the stable
`[HDF4_RUNTIME_UNAVAILABLE]` message naming the detected cause, the
capabilities that still work, and the fix — never an unexplained import
traceback. An audit of an archive containing HDF4 files still completes,
recording those files as `unsupported` with the same code. For HDF4 on
Windows use the conda-forge install above (or **WSL** with the pip
instructions); a dedicated CI job pins this degraded-capability contract on
every change.

## Command line usage

```console
# Recompress a netCDF4 file in place (verified before replacement).
ncarnate observations.nc --complevel 9

# Keep the original; write observations_recompressed.nc beside it.
ncarnate --no-overwrite observations.nc

# Convert an HDF-EOS2 granule -> granule.nc with CF geolocation.
ncarnate AMSR_E_L3_SeaIce12km_B02_20020619.hdf

# Convert the raw SDS payload only (unsupported-projection escape hatch).
ncarnate --no-geolocation granule.hdf

# Recurse over a directory tree.
ncarnate -r /data/archive
```

Exit codes: `0` success, `1` one or more files failed, `2` bad input paths or
arguments.

## Audit an archive in 5 minutes

Before converting a terabyte archive, run a **read-only audit**: it never opens
science arrays, never touches the network, and never writes to the files it
inspects. It discovers files, detects formats, inspects metadata, classifies
each file into a readiness taxonomy, and prints a summary by files *and* bytes.

```console
# Assess an archive (recursive, read-only) and print a readiness summary.
ncarnate audit /data/archive

# Write the per-file migration manifest (JSONL is the contract; .csv gives a
# flat spreadsheet view). Add --checksum sha256 for a manifest you intend to
# execute later.
ncarnate audit /data/archive --output manifest.jsonl --checksum sha256
```

Each JSONL line is one versioned, schema-validated file record — path,
checksum, status, blockers, and the conversion plan — designed so a later
`ncarnate convert --manifest` (and every downstream tool) consumes it unchanged.
The bare `ncarnate <path>` and `ncarnate convert <path>` forms are unchanged.

## Convert exactly what the audit blessed

The golden path for an archive migration is two steps: **audit an archive, then
convert exactly what it blessed.** `convert --manifest` executes the audit's
manifest — it re-verifies each granule's recorded `sha256` before touching it,
converts only the statuses you select (`ready` by default), writes a mirrored
output tree, and **never modifies a source** unless you pass `--in-place`.

```bash
# 1. Audit the archive, recording a per-file sha256 in the manifest.
ncarnate audit /data/archive --output manifest.jsonl --checksum sha256

# 2. Convert exactly the `ready` granules into a mirrored ./modern tree.
#    --root anchors reads to a directory you control (the manifest is untrusted
#    input, so its recorded root is not trusted as the read base by default;
#    pass --allow-manifest-root to opt into trusting it instead). A record whose
#    bytes changed since the audit (sha256 mismatch) is skipped with an error;
#    a blocker is never converted; sources are left untouched.
ncarnate convert --manifest manifest.jsonl --out-dir ./modern --root /archive

# Widen the selection once you've read the report; resume an interrupted run.
ncarnate convert --manifest manifest.jsonl --out-dir ./modern --root /archive \
    --status ready,already_modern --skip-existing
```

The end-of-run summary counts converted / skipped / failed with reasons, and the
exit code is non-zero **iff** a selected record failed — so a partial failure on
a terabyte run surfaces loudly instead of silently mis-converting.

**Destination collision preflight.** Before any directory or output file is
created, every selected record's destination is computed up front — from the
source's *detected bytes*, never the manifest's declared format — and any
collision refuses the **entire run** with exit code 2 and a stable
`[DESTINATION_COLLISION]` message on stderr listing every involved source and
the contested destination. No last-writer-wins, no auto-rename, no partial
proceed, and nothing is written. Refused collisions include: two records
landing on one output path (e.g. an `a.hdf` → `a.nc` conversion next to a real
`a.nc` sibling), case-fold-equivalent names (one file on NTFS/APFS), duplicate
records for one source, an output tree overlapping a source tree (symlinks
resolved), and a pre-existing destination unless you pass `--skip-existing` to
resume.

## Library usage

```python
from ncarnate import (
    recompress, audit_path, AuditOptions, convert_manifest, ConvertOptions,
)

# Lossless recompression; returns the output path.
recompress("observations.nc", complevel=9)

# HDF-EOS2 conversion; the .hdf source is never replaced.
recompress("granule.hdf", dst="granule.nc")

# Read-only archive audit; returns an AuditReport (report.summary, report.files).
report = audit_path("/data/archive", AuditOptions(recursive=True))

# Execute an audit manifest; returns a ConvertResult (converted/skipped/failed).
result = convert_manifest("manifest.jsonl", ConvertOptions(out_dir="./modern"))
```

## Example

The AMSR-E daily 12.5 km sea-ice granule this project grew up around:

| File | Input | Output |
| --- | --- | --- |
| netCDF4 recompression (`--complevel 9`) | 42.6 MB | 19.9 MB |
| HDF-EOS2 → netCDF4 (+ reconstructed lat/lon) | 60.2 MB | 35.5 MB |

Both outputs re-read value-identically to their sources; the conversion
additionally carries CF `polar_stereographic` grid mappings and coordinates for
both hemispheric grids. The northern grid's reconstructed latitudes/longitudes
agree with The HDF Group's independent conversion of the same granule to within
10⁻⁵ degrees (about a metre), the tolerance the test suite enforces.

## Supported inputs

- **netCDF4 / HDF5** and **netCDF3** — recompressed via the netCDF4 library.
- **HDF4 / HDF-EOS2** — read via the pyhdf SD API. GRID structures with GCTP
  polar-stereographic, geographic, and Lambert-azimuthal (EASE-Grid)
  projections; SWATH structures with direct or dimension-mapped geolocation.
  Output is always netCDF4 — HDF4 is never written.

## Development

```console
pip install -e ".[test]"
ruff check .
pytest
```

The test suite runs entirely offline against small committed fixtures trimmed
from real granules (provenance sidecars included); cross-checks against the raw
multi-MB granules self-skip where the local granule store is absent.

## License

MIT — see [LICENSE](https://github.com/ErickShepherd/ncarnate/blob/main/LICENSE).
Built by [Erick Shepherd](https://erickshepherd.com).
