Metadata-Version: 2.4
Name: nem-catalog
Version: 0.1.0
Summary: Machine-readable URL catalog for AEMO NEMWEB (JSON + JSON Schema + Python SDK)
Project-URL: Homepage, https://github.com/ZhipengHe/nem-catalog
Project-URL: Repository, https://github.com/ZhipengHe/nem-catalog
Project-URL: Catalog, https://zhipenghe.me/nem-catalog/catalog.json
Project-URL: Schema, https://zhipenghe.me/nem-catalog/catalog.schema.json
Project-URL: Changelog, https://github.com/ZhipengHe/nem-catalog/blob/master/CHANGELOG.md
Author-email: Zhipeng He <zippo.he@qut.edu.au>
License: MIT
License-File: LICENSE
Keywords: aemo,australian-energy-market,data-catalog,electricity-market-data,json-schema,nem,nemweb
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: jsonschema>=4.20; extra == 'dev'
Requires-Dist: mypy>=1.20; extra == 'dev'
Requires-Dist: pre-commit>=4.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Requires-Dist: ruff>=0.15; extra == 'dev'
Description-Content-Type: text/markdown

# nem-catalog — Machine-readable URL catalog for AEMO NEMWEB

[![Catalog](https://img.shields.io/badge/catalog-live-blue)](https://zhipenghe.me/nem-catalog/catalog.json)
[![Schema](https://img.shields.io/badge/schema-v1.0.0-green)](https://zhipenghe.me/nem-catalog/catalog.schema.json)
[![PyPI](https://img.shields.io/pypi/v/nem-catalog)](https://pypi.org/project/nem-catalog/)
[![License: MIT](https://img.shields.io/badge/license-MIT%20%2B%20CC0-brightgreen)](LICENSE)
[![Last successful crawl](https://img.shields.io/badge/last%20crawl-TBD-lightgrey)](https://github.com/ZhipengHe/nem-catalog/actions/workflows/weekly-refresh.yml)

A versioned JSON catalog + JSON Schema that maps `(NEMWEB dataset key, time range) → candidate URLs`, covering all four NEMWEB repositories (`Reports`, `MMSDM`, `NEMDE`, `FCAS_Causer_Pays`). Released under MIT (code) and CC0 (catalog data).

## Quick start — no install required

```bash
curl -s https://zhipenghe.me/nem-catalog/catalog.json \
  | jq '.datasets["Reports:DispatchIS_Reports"].tiers.ARCHIVE'
```

Output:

```json
{
  "path_template": "/Reports/ARCHIVE/DispatchIS_Reports/",
  "filename_template": "PUBLIC_DISPATCHIS_{date}.zip",
  "filename_regex": "^PUBLIC_DISPATCHIS_\\d{8}\\.zip$",
  "example": "PUBLIC_DISPATCHIS_20250407.zip",
  "cadence": "daily_rollup"
}
```

Build the full URL: `https://nemweb.com.au` + `path_template` + `filename_template` (with `{date}` substituted as `yyyymmdd`). Placeholder vocabulary is in the catalog's top-level `placeholders` field.

## Stability

v0.1 is **experimental**. API may change before v1.0. For reproducible research, pin the catalog version:

```python
catalog = nem_catalog.fetch_latest(catalog_version="2026.04.18")
```

## Python usage

```bash
pip install nem-catalog
```

```python
import nem_catalog

# Primary (library-pure, deterministic):
catalog = nem_catalog.load("catalog.json")

urls = catalog.resolve(
    "Reports:DispatchIS_Reports",
    from_="2025-04-01",
    to_="2025-04-02",
)
# → list of candidate URLs. Caller is responsible for reachability.

# Convenience (live fetch + cache + fallback):
catalog = nem_catalog.fetch_latest()

# Preview cardinality before materializing:
n = catalog.count("Reports:DispatchIS_Reports", from_="2024-01-01", to_="2024-12-31")
```

> **Expected `UserWarning`:** `Reports:*` datasets with both an ARCHIVE and a
> rolling CURRENT tier emit a one-line warning when you query historical
> (ARCHIVE-era) dates. The SDK is telling you the live tier has no data that
> old, so it routed to ARCHIVE. The returned URLs are correct.

### Not every dataset resolves to concrete URLs in v0.1

> **Coverage in v0.1:** roughly **1 in 6** of the 362 dataset keys resolve
> cleanly today (~16%, mostly `Reports:*` ARCHIVE tiers). The remaining ~84%
> raise `NonResolvableTemplateError` — including almost all `MMSDM:*` tables
> (file-sequence suffix `{d2}`/`{nn}`) and every live CURRENT tier (16-digit
> publish ID `{aemo_id}`).
>
> Per repo: `Reports` 53/96 (55%), `MMSDM` 4/259 (~2%), `NEMDE` 2/6,
> `FCAS_Causer_Pays` 0/1. v0.2 will add `list_urls()` for the non-temporal
> cases by reading NEMWEB directory listings.

`resolve()` only returns URLs when the tier's filename template can be built
from a date range alone. AEMO filenames in rolling CURRENT tiers often embed a
participant ID (e.g. `{aemo_id}`) or a file-sequence suffix (e.g. `{nn}`) that
the SDK cannot compute without extra input. For those, `resolve()` raises
`NonResolvableTemplateError` rather than return a broken URL string.

```python
# Raises NonResolvableTemplateError — CURRENT filename has {aemo_id}
catalog.resolve("Reports:DispatchIS_Reports", from_="2026-04-17", to_="2026-04-18")

# Works — ARCHIVE filename is pure temporal
catalog.resolve("Reports:DispatchIS_Reports", from_="2025-04-01", to_="2025-04-02")
```

Inspect the raw template for any dataset via `catalog.datasets[key]['tiers']`
and build the URL yourself, or pin the query to an ARCHIVE-covered date range.
A future release will add an enumeration API for these datasets.

## Not for you if...

- You want a **pandas DataFrame** of NEMWEB data → use [NEMOSIS](https://github.com/UNSW-CEEM/NEMOSIS). It's the production-grade Python pipeline for researchers.
- You want **forecast data** (pre-dispatch, PASA) → use [NEMSEER](https://github.com/UNSW-CEEM/NEMSEER).
- You want **emissions data** → use [NEMED](https://github.com/UNSW-CEEM/NEMED).

`nem-catalog` serves the layer *below* these tools: a shared metadata + canonical JSON shape describing NEMWEB's URL grammar. Non-Python consumers (R, Julia, shell) can use the JSON directly without installing anything.

## Shell cookbook (R/Julia/shell users)

See [`docs/cookbook.md`](docs/cookbook.md) for recipes including URL expansion, date iteration, and parallel download with `xargs`.

## How it's built

See [`docs/architecture.md`](docs/architecture.md). Briefly: `extract_patterns.py` mirrors NEMWEB directory listings weekly, derives URL patterns, and a hybrid auto+curated merge produces `catalog.json`. Weekly GitHub Actions runs the whole pipeline and opens a PR on diffs.

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md).

## License

- Code: MIT. See [`LICENSE`](LICENSE).
- Catalog JSON: CC0 (public domain).
