Metadata-Version: 2.4
Name: openeolib
Version: 0.2.2
Summary: Python library for Earth observation data providers, geospatial analysis, plume simulation, and reusable visualization
Author: tkxu
License: Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: matplotlib>=3.7
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.12
Provides-Extra: provider
Requires-Dist: openeo>=0.26; extra == "provider"
Requires-Dist: rasterio>=1.3; extra == "provider"
Requires-Dist: xarray>=2023.1; extra == "provider"
Requires-Dist: netCDF4>=1.6; extra == "provider"
Provides-Extra: era5
Requires-Dist: cdsapi>=0.6; extra == "era5"
Requires-Dist: xarray>=2023.1; extra == "era5"
Requires-Dist: netCDF4>=1.6; extra == "era5"
Provides-Extra: jma
Requires-Dist: xarray>=2023.1; extra == "jma"
Requires-Dist: cfgrib>=0.9.10; extra == "jma"
Requires-Dist: dask>=2023.1; extra == "jma"
Requires-Dist: pyyaml>=6.0; extra == "jma"
Requires-Dist: JMA-grib2>=0.0.4a3; extra == "jma"
Provides-Extra: himawari
Requires-Dist: xarray>=2023.1; extra == "himawari"
Requires-Dist: satpy>=0.55; extra == "himawari"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: flake8>=6.1; extra == "dev"
Requires-Dist: mypy>=1.5; extra == "dev"
Dynamic: license-file

# OpenEO-LIB

OpenEO-LIB is a Python library for satellite-based Earth observation workflows, including data providers, geospatial analysis, plume simulation, and reusable visualization components.

The package is designed so that reusable library functionality is kept separate from application-specific validation and research code.

## Current architecture

```text
openeolib/
├── __init__.py
├── analyzer.py          Analysis pipeline orchestration
├── animation.py         Generic scalar-field animation (GridAnimationEngine)
├── engines.py           Inference-engine interfaces and built-ins
├── eo_cache.py          Persistent scientific-data cache (ScienceCacheStore)
├── eo_types.py          Shared data structures and type definitions
├── eo_utils.py          Geospatial and wind utilities
├── panels.py            Generic Matplotlib visualization panels
├── report.py            Generic figure/report composition
├── roc.py               ROC construction
├── simulator.py         Synthetic plume generation
├── theme.py             Theme definition only (Theme, DEFAULT_THEME)
└── providers/
    ├── base.py               BaseProvider ABC (raw Dataset retrieval)
    ├── provider.py           AnalyzerProvider / EOProvider (S2+ERA5+S5P integration)
    ├── openeo_client.py      openEO connection client
    ├── _http_utils.py        Shared retry/local-file-cache helpers
    ├── _protocols.py         Cross-provider provenance accessor (get_provenance())
    ├── sentinel2.py          Sentinel-2 L2A band retrieval
    ├── sentinel5p.py         Sentinel-5P L2 CH4 column retrieval
    ├── era5.py               ERA5 wind/pressure/temperature retrieval
    ├── msm.py                JMA MSM (mesoscale model) wind retrieval
    ├── radar_gpv_client.py   JMA nationwide composite radar GPV download
    ├── rain.py               JMA rainfall / XRAIN providers + animate_rain()
    ├── amedas.py             JMA AMeDAS surface pressure/temperature retrieval
    ├── radiosonde.py         University of Wyoming radiosonde sounding retrieval
    ├── pwv.py                GPT3 / VMF3 blind-grid PWV retrieval
    ├── himawari.py           Himawari-8/9 AHI cloud-top brightness temperature
    └── _s2_utils.py          Sentinel-2 coordinate/time conversion utilities


```

Provider responsibilities are deliberately separated. `BaseProvider` and
`AnalyzerProvider` are independent interfaces with different responsibilities;
`AnalyzerProvider` does not extend `BaseProvider`. `BaseProvider` is the minimal
file/data-opening interface for providers that return an `xarray.Dataset`
without application-level analysis. `AnalyzerProvider` is the higher-level
interface used by `EOAnalyzer` and returns an `ObservationBundle` for a site and
observation time -- a `typing.Protocol` (structural typing), so a provider
conforms by having a matching `get_bundle(site, dt)` method, not by explicitly
inheriting from it. `EOProvider` implements it
for the openEO/Sentinel-2/Sentinel-5P workflow and contains the
workflow-specific quality and time-alignment logic. `AnalyzerProvider`/
`ObservationBundle` is *not* a general-purpose provider contract -- it is
`EOAnalyzer`'s own methane/plume-detection input shape (it reads `bands["B11"]`/`["B12"]`
directly), produced only by `EOProvider` and `PlumeSimulator`.

For code that wants to treat any provider's result uniformly regardless of its
own return shape (`xr.Dataset`, a `Dict`, or `None`) -- e.g. for logging or an
audit trail -- `providers._protocols.get_provenance(result)` extracts a
normalized `{source, provider_class, datetime_utc, provenance_key}` dict from
whichever of the three metadata conventions this package's providers happen to
use (`.attrs["source"]`/`["provider_class"]`, `.attrs["provider"]`, or dict
keys `"backend"`/`"datetime_utc"`). It requires no changes to any existing
provider.

The visualization layer (`panels.py`, `theme.py`, `report.py`, `animation.py`)
is kept domain-agnostic. Domain providers may know the schema and units of
their own source data, but they do not depend on application-specific
research pipelines.

Likewise, `theme.py` contains only the `Theme` data structure and `DEFAULT_THEME`.

## Installation

```bash
pip install openeolib
```

Development dependencies are optional and are intended for the private test suite:

```bash
pip install -e ".[dev]"
```

Optional provider dependencies are grouped by feature:

```bash
pip install -e ".[provider]"
pip install -e ".[era5]"
pip install -e ".[jma]"
pip install -e ".[himawari]"
```

The core package supports Python 3.9 or later. Provider-specific optional dependencies may have their own Python-version requirements.

The JMA extra (`openeolib[jma]`) is required for `MSMProvider`. The module
`openeolib.providers.msm` remains importable without the optional dependencies,
but constructing `MSMProvider` raises a clear `ImportError` listing the missing
dependencies and the installation command. This keeps optional JMA support from
breaking the core package import.

## Public visualization API

The reusable visualization API consists of five panels:

- `VectorFieldPanel` — arbitrary geospatial vector fields with optional scalar contours.
- `BasemapPanel` — an already prepared RGB image with an optional marker.
- `RawBandsPanel` — arbitrary grids of 2D images with a shared scale.
- `ScalarMapPanel` — arbitrary 2D scalar fields.
- `DetectionMaskPanel` — boolean, probability, or coverage masks.

Example:

```python
import matplotlib.pyplot as plt
import numpy as np
from openeolib import ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

fig = plt.figure(figsize=(6, 5))
gs = fig.add_gridspec(1, 1)[0]
ScalarMapPanel().draw(
    fig,
    gs,
    field,
    title="Scalar field",
    axis_mode="none",
)
fig.savefig("scalar_field.png", dpi=150, bbox_inches="tight")
```

### Vector fields

`VectorFieldPanel` is intentionally not ERA5-specific. It accepts latitude/longitude grids and arbitrary vector components. A caller may supply any bounding box through `extent=(west, east, south, north)`.

```python
from openeolib import VectorFieldPanel

panel = VectorFieldPanel(quiver_stride=3)
panel.draw(
    fig,
    gs,
    lats=lats,
    lons=lons,
    u=u,
    v=v,
    extent=(120, 150, 20, 50),
    title="Wind field",
)
```

There is no Japan-specific bounding-box constant in the library.

### Theme

Use `Theme` when a caller needs to customize the visual appearance:

```python
from openeolib import Theme

light = Theme(
    bg="#ffffff",
    panel_bg="#ffffff",
    grid_color="#cccccc",
    text_primary="#222222",
)
```

`Theme` does not contain plotting operations or application-specific labels, flags, or data extraction rules.

## Generic reports

`SiteReportBuilder` composes caller-supplied panels into a single-site report Figure. It does not know about methane, ROC curves, quality flags, or a particular inference engine -- it only handles grid layout, the report title, and file saving.

Each entry in `panels` is `(label, factory, height_ratio)`. `factory` is called once with the report's `Theme` and must return a `(fig, gs) -> None` draw function; any panel-specific data (the field to plot, its title, ...) is bound into that closure by the caller, not inspected by `SiteReportBuilder` itself.

```python
import numpy as np
from openeolib import SiteReportBuilder, ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

report = SiteReportBuilder(
    panels=[
        (
            "Scalar field",
            lambda theme: lambda fig, gs: ScalarMapPanel(theme=theme).draw(
                fig, gs, field, title="Scalar field", axis_mode="none",
            ),
            1.0,
        ),
    ],
    report_title="Example report",
)

report.build_site({"site": {"id": "SITE-01"}}, save_path="report.png")
```

Application-specific validation reports can be built with the components under `examples/validation_panels.py` and `examples/validation_report.py` without adding those domain concepts to the reusable package.

## Analysis and simulation

The main public analysis components include:

```python
from openeolib import EOAnalyzer, PlumeSimulator, InferenceEngine
```

`EOAnalyzer` coordinates provider data and an inference engine. `PlumeSimulator` provides synthetic plume data for testing and demonstrations. `InferenceEngine` defines the interface for custom detection/quantification algorithms.

## Providers

The package contains provider implementations for openEO, Sentinel-2, Sentinel-5P, ERA5, MSM, JMA radar rainfall products, JMA AMeDAS surface observations, University of Wyoming radiosonde soundings, GPT3/VMF3 blind-grid PWV, and Himawari-8/9 cloud-top brightness temperature. Provider-specific dependencies are optional where practical.

`XrainProvider` accepts NetCDF and CSV input. If a CSV has no time column, it
creates a single `time` coordinate containing `NaT` rather than inventing an
observation timestamp. Callers that require a real observation time must provide
a time column or pass `time_name=`. This distinction is intentional and prevents
silent fabrication of temporal metadata.

`RadarGpvClient` is a public retrieval utility, not a `BaseProvider` or `AnalyzerProvider`.
It resolves JMA nationwide composite radar GPV archive URLs, downloads the archive,
and extracts the target GRIB2 file. `JmaRainProvider` is the dataset-opening provider
that parses those extracted files into an `xarray.Dataset`. This separation keeps
network/archive handling distinct from dataset parsing.

`ERA5Provider` supports surface fields and pressure-level wind processing, including height-aware interpolation for wind products. Provider modules return data; visualization remains a separate concern.

`EOProvider` (the S2/ERA5/S5P integration used by `EOAnalyzer`) persists every
`fetch()` result on disk (`ScienceCacheStore`, `enable_cache=True` by default),
keyed by `(lat, lon, dt, band_set, radius_km, enable_100m_wind)`. An outright
failure (no S2 scene at all) is never cached, so a later call retries instead
of replaying the same empty result forever.

`MSMProvider` retrieves JMA MSM GPV wind data and requires the `jma` extra. The
public facade exposes `MSMProvider`, `MSM_AVAILABLE`, and
`MSM_MISSING_DEPENDENCIES` so applications can detect optional support without
catching an import failure from the core package.

`AMeDASProvider` and `RadiosondeProvider` retrieve historical JMA surface
observations and University of Wyoming upper-air soundings respectively, both
by scraping the providers' public data-search pages (`requests` +
`beautifulsoup4`) rather than an official API, and both cache retrieved
results indefinitely on disk (`ScienceCacheStore`), since historical
observations never change. Both return `xr.Dataset` (`time`-dimensioned for
AMeDAS; `time`/`level`-dimensioned, NaN-padded across variable-length
profiles, for radiosonde soundings), matching the convention used by the
rest of the providers rather than a bespoke `Dict` shape.

`PWVProvider` (GPT3 blind climatological grid) and `VMF3Provider` (VMF3_OP
per-epoch NWP grid), in `providers/pwv.py`, are independent, GNSS-external
sources of Zenith Hydrostatic/Wet Delay for validating or cross-checking an
ERA5-based PWV pipeline. Both ultimately produce a PWV estimate via the
Bevis et al. (1994) Pi(Tm) conversion; see the module docstring for the
K1/K2 refractivity-constant caveat when comparing against another pipeline's
convention.

`HimawariCloudClient` / `HimawariCloudProvider`, in `providers/himawari.py`,
download and decode Himawari-8/9 AHI gridded cloud-top brightness temperature
from CEReS (Chiba University), requiring the `himawari` extra (`satpy`).
**This module's underlying data has its own license, separate from
openeolib's Apache-2.0 code license: non-commercial/research use only, no
redistribution, and required attribution when used or published.** Both
classes emit a `UserWarning` with the attribution text on instantiation; see
the module docstring for the full terms before using it beyond research.

For ERA5, CDS credentials and the current CDS API/client configuration are required for live retrieval. Unit tests mock retrieval where network access is unnecessary.

## Animation

`GridAnimationEngine` in `openeolib.animation` is the generic animation component. Domain-specific wrappers, such as `animate_rain()` in `providers.rain`, configure rainfall-specific variables and color scales before delegating rendering to the generic engine.

```python
from openeolib.animation import GridAnimationEngine

engine = GridAnimationEngine()
engine.animate_scalar_field(
    grids=grids,
    lats=lats,
    lons=lons,
    timestamps=timestamps,
    output_path="animation.gif",
)
```

## Validation examples

Validation-only components are kept outside the package:

```python
from examples.validation_panels import (
    SpectralPanel,
    StatisticalPanel,
    FlagsPanel,
)
from examples.validation_report import ValidationReportBuilder
```

These modules are useful for project-specific evaluation but are not exported from `openeolib` and should not be treated as stable library APIs.

## Testing

The test suite is maintained for internal development only. `tests/` is
intentionally excluded from both the published package (see `MANIFEST.in`)
and this repository's git history (see `.gitignore`), so it is not available
to clone or install. This has no bearing on installing or using the library.

## License

OpenEO-LIB is distributed under the Apache License 2.0. See `LICENSE` for the full license text.
