{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "emix-001",
   "metadata": {},
   "source": [
    "# Updating the electricity supply and trade mix\n",
    "\n",
    "This workflow builds a database whose electricity **supply mix** (generation by\n",
    "technology) and **trade mix** (bilateral sourcing between regions) reflect\n",
    "recent statistics. Both updates rewrite the *composition* of the table while\n",
    "preserving every column total: within each region the weight only moves\n",
    "*between* the electricity labels, so every buyer keeps its total electricity\n",
    "input and the rest of the table is untouched.\n",
    "\n",
    "The shares come from two open data providers:\n",
    "\n",
    "* **[EMBER](https://ember-energy.org/)** — generation-by-fuel, used for the\n",
    "  **supply** mix. EMBER covers every country worldwide.\n",
    "* **[ENTSO-E](https://transparency.entsoe.eu/)** — scheduled cross-border\n",
    "  commercial exchanges, used for the **trade** mix. Bilateral flows only exist\n",
    "  for the European synchronous area, which is exactly where a trade mix is\n",
    "  meaningful.\n",
    "\n",
    "The workflow applies to any database that exposes a **disaggregated power\n",
    "sector** (separate generation technologies rather than a single electricity\n",
    "sector): EXIOBASE (hybrid and monetary, IOT and SUT) and EMERGING-E (IOT). The\n",
    "examples below parse **EXIOBASE Hybrid v3.3.18** as a SUT and go end to end:\n",
    "parse → supply mix → pool the electricity trade → trade mix → export."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-002",
   "metadata": {},
   "source": [
    "## Core methods\n",
    "\n",
    "* `Database.update_supply_mix` — rewrite the generation mix in place (from\n",
    "  EMBER, or from explicit shares);\n",
    "* `Database.pool_trade` — prepare a SUT so the technology mix and the trade mix\n",
    "  become two independent market-share columns;\n",
    "* `Database.update_trade_mix` — rewrite the regional sourcing of a traded item\n",
    "  per destination market (from ENTSO-E, or from explicit shares);\n",
    "* `Database.get_mix` — read back a region-by-sector production mix (IOT).\n",
    "\n",
    "`Database.update_supply_mix_iot` and `Database.update_mix_iot` are kept as\n",
    "backward-compatible IOT-only aliases of `update_supply_mix`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-003",
   "metadata": {},
   "source": [
    "## Configuring the data-provider API keys\n",
    "\n",
    "Passing the special string `\"electricity\"` to `update_supply_mix` /\n",
    "`update_trade_mix` makes MARIO derive the shares from the provider data: EMBER\n",
    "generation for the supply mix, the ENTSO-E import mix for the trade mix.\n",
    "\n",
    "MARIO ships a **reduced snapshot** of each provider inside the package, so both\n",
    "methods run **offline with no key** — the reproducible fallback, frozen at the\n",
    "snapshot vintage. To fetch a specific or more recent `year` **live**, register\n",
    "a free key and let MARIO query the provider API:\n",
    "\n",
    "* EMBER — register at https://ember-energy.org/data/api/\n",
    "* ENTSO-E — request the API token from https://transparency.entsoe.eu/\n",
    "\n",
    "Keys can be provided four ways; the first one found wins, so a secret never has\n",
    "to live in code:\n",
    "\n",
    "1. inline at the call site, `api_key={\"ember\": \"<key>\"}`;\n",
    "2. `mario.set_api_keys(ember=\"...\", entsoe=\"...\")`, set once per session;\n",
    "3. the `EMBER_API_KEY` / `ENTSOE_API_KEY` environment variable;\n",
    "4. the git-ignored `mario/settings/api_keys.yaml` file (copy\n",
    "   `api_keys.example.yaml`; **never commit real keys**).\n",
    "\n",
    "In a shared notebook, read the keys from the environment rather than typing\n",
    "them in a cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import mario\n",
    "\n",
    "# Configure the provider keys once for the session. Reading them from the\n",
    "# environment keeps the actual secrets out of the notebook.\n",
    "mario.set_api_keys(\n",
    "    ember=os.environ.get(\"EMBER_API_KEY\"),\n",
    "    entsoe=os.environ.get(\"ENTSOE_API_KEY\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-005",
   "metadata": {},
   "source": [
    "Once the keys are configured you refer to a provider by name at the call site\n",
    "(`api_key=\"ember\"` / `api_key=\"entsoe\"`). Omit `api_key` entirely to use the\n",
    "packaged snapshot instead of a live fetch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-006",
   "metadata": {},
   "source": [
    "## Parse a database\n",
    "\n",
    "Download the EXIOBASE Hybrid v3.3.18 files from Zenodo and parse the SUT. The\n",
    "hybrid table ships the disaggregated electricity generation activities and\n",
    "commodities that the mix updates need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "info = mario.download_hybrid_exiobase(path=\"/path/to/3.3.18\", table=\"SUT\")\n",
    "\n",
    "db = mario.parse_exiobase(\n",
    "    path=\"/path/to/3.3.18\",\n",
    "    table=\"SUT\",\n",
    "    unit=\"Hybrid\",\n",
    "    extensions=\"all\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-008",
   "metadata": {},
   "source": [
    "## Update the supply mix from EMBER\n",
    "\n",
    "`update_supply_mix(\"electricity\")` rewrites the generation mix of one scenario\n",
    "in place. The EMBER taxonomy is coarser than the disaggregated electricity\n",
    "labels, so MARIO:\n",
    "\n",
    "1. aggregates the database electricity bundle to the compatible EMBER fuel\n",
    "   groups (coal, gas, nuclear, hydro, wind, solar, bioenergy, other renewables,\n",
    "   other fossil);\n",
    "2. reads the EMBER generation shares for the requested `year`;\n",
    "3. redistributes each group total back to the original database labels using\n",
    "   the **current internal composition** of each group.\n",
    "\n",
    "On a SUT, MARIO first aggregates the disaggregated electricity commodities into\n",
    "one shared `electricity` commodity — using the packaged profile, no mapping\n",
    "required for EXIOBASE-style layouts — and rewrites the market shares of the\n",
    "generation activities in the supply block `s`. Transmission and distribution\n",
    "labels are left unchanged. On an IOT the generation sectors are redistributed\n",
    "across the `z` and `Y` blocks instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Live EMBER fetch for a specific year (uses the key configured above);\n",
    "# omit api_key to read the packaged snapshot instead.\n",
    "db.update_supply_mix(\"electricity\", scenario=\"baseline\", year=2023, api_key=\"ember\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-010",
   "metadata": {},
   "source": [
    "Regions that EMBER does not cover, or that report no positive generation, are\n",
    "left unchanged and reported in the logs. When a covered region has no\n",
    "observation for the requested `year`, MARIO falls back to the nearest available\n",
    "year for that region.\n",
    "\n",
    "**Inspecting and customising the mix.** A few common variations:\n",
    "\n",
    "* **Explicit shares** — instead of `\"electricity\"`, pass an explicit\n",
    "  `region -> {label: share}` mapping to move weight between named labels\n",
    "  (`rescale=True` normalises arbitrary positive totals). On an IOT database\n",
    "  `get_mix(\"electricity\")` reads the current mix back as\n",
    "  `region -> {sector: share}`.\n",
    "* **Aggregated regions** — EXIOBASE Rest-of-World regions and user aggregations\n",
    "  are expanded to their member countries automatically; pass\n",
    "  `region_aggregation=...` to supply the concordance explicitly.\n",
    "* **`aggregate_as_ember=True`** — collapse the disaggregated electricity labels\n",
    "  to the EMBER fuel groups after the update.\n",
    "* **Column selectors** — `column_regions`, `column_sectors` and\n",
    "  `column_categories` restrict the rewrite to a subset of buyer/demand columns.\n",
    "\n",
    "See `Database.update_supply_mix` in the API reference for the full argument\n",
    "list. The diagrams below show where the weight lands for each table layout.\n",
    "\n",
    "![Supply mix on IOT and on a SUT table: the sector/commodity bundle rows are redistributed across the z/Y and u/Yc buyer columns, within each region.](../../_static/images/update_supply_mix_iot_sut.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-011",
   "metadata": {},
   "source": [
    "> **Chenery-Moses tables: update the mix on every destination market.**\n",
    "> On tables converted with `to_chenery_moses`, the supply block entangles\n",
    "> technology and trade: one region's technology mix is replicated — scaled by\n",
    "> its trade shares — inside the `s` columns of every destination it exports to.\n",
    "> The default `column_regions=None` correctly rewrites the region's rows in\n",
    "> **all** destination markets, preserving each destination's trade share.\n",
    "> Restricting `column_regions` to the domestic market would leave the exported\n",
    "> production on the old mix (one fleet, two mixes — physically inconsistent).\n",
    "> Isard tables and pooled tables built with `pool_trade` are immune by\n",
    "> construction, because the technology mix is stored once."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-012",
   "metadata": {},
   "source": [
    "## Pool the electricity trade\n",
    "\n",
    "On a raw Isard SUT the bilateral sourcing lives in the use-side rows, so every\n",
    "buyer of a region can have its own origin split and the technology mix and the\n",
    "trade mix are entangled. `update_supply_mix(\"electricity\")` above already\n",
    "aggregated the disaggregated generation commodities into one commodity named\n",
    "`electricity`; `pool_trade` pools that commodity, adding per region one\n",
    "`\"electricity - supply\"` pass-through activity and one `\"electricity - need\"`\n",
    "market commodity:\n",
    "\n",
    "1. the pass-through activity consumes the whole domestic output of electricity;\n",
    "2. every buyer moves onto the domestic need commodity, keeping its total input;\n",
    "3. the supply block routes each destination market to the origin pass-through\n",
    "   activities with the **bilateral flows observed on the Isard use side**.\n",
    "\n",
    "Destination-level trade totals are preserved exactly, and the initial market\n",
    "shares equal the observed origin shares. Afterwards the technology mix (on the\n",
    "`electricity` commodity column) and the trade mix (on the `electricity - need`\n",
    "column) live in two separate market-share columns of `s` and can be updated\n",
    "independently.\n",
    "\n",
    "![`pool_trade` adds, per region, one \"- supply\" pass-through activity and one \"- need\" market commodity: buyers move onto the domestic need commodity and the supply block routes each destination market to the origin supply activities, so the technology mix and the trade mix become two separate market-share columns of s.](../../_static/images/pool_trade.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "db.pool_trade(\"electricity\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-014",
   "metadata": {},
   "source": [
    "## Update the trade mix from ENTSO-E\n",
    "\n",
    "`update_trade_mix(\"electricity\")` reads the ENTSO-E import mix — a first-order\n",
    "**net commercial** decomposition of scheduled exchanges (ENTSO-E A09), not\n",
    "physical cross-border flows — and rewrites each destination's electricity\n",
    "sourcing. On the pooled table the traded rows live in the supply block `s`, so\n",
    "this is an **Activity-level** mix: pass `level=\"Activity\"` and name the pooled\n",
    "pair through `items` (the `- supply` pass-through activity) and `commodities`\n",
    "(the `- need` market column). Read the exact labels back from\n",
    "`db.meta.pooled_trade_map`: `update_supply_mix` aggregated the disaggregated\n",
    "generation commodities into one commodity named `electricity` (lowercase), so\n",
    "the pooled pair is `electricity - supply` / `electricity - need`. ENTSO-E\n",
    "supplies only the shares.\n",
    "\n",
    "`fill_uncovered_domestic=True` sets the regions ENTSO-E does not cover\n",
    "(non-European: US, CN, JP, ... and the RoW aggregates, which are electrically\n",
    "near-isolated) to domestic-only. Leave it `False` to keep them on their\n",
    "original sourcing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-015",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Live ENTSO-E fetch into a new scenario cloned from the supply-mix baseline.\n",
    "# Read the exact pooled labels back from the map pool_trade recorded, rather\n",
    "# than hard-coding them.\n",
    "pooled = db.meta.pooled_trade_map[\"electricity\"]\n",
    "\n",
    "db.update_trade_mix(\n",
    "    \"electricity\",\n",
    "    items=pooled[\"supply\"],\n",
    "    commodities=pooled[\"need\"],\n",
    "    level=\"Activity\",\n",
    "    scenario=\"electricity_mix\",\n",
    "    clone_from=\"baseline\",\n",
    "    year=2023,\n",
    "    api_key=\"entsoe\",\n",
    "    fill_uncovered_domestic=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-016",
   "metadata": {},
   "source": [
    "The trade mix covers the ENTSO-E (European) area only; non-EU zones with poor\n",
    "coverage are kept as-is. The basis was validated against Electricity Maps,\n",
    "whose data is proprietary and cannot be redistributed.\n",
    "\n",
    "The `update_trade_mix` API reference documents the explicit\n",
    "`destination -> {origin: share}` form (for arbitrary items and non-electricity\n",
    "trade), the `column_sectors` / `column_categories` selectors, and the block\n",
    "each table layout rewrites."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-017",
   "metadata": {},
   "source": [
    "## Export\n",
    "\n",
    "Write the updated scenario to MARIO's text format (or `to_excel` / `to_parquet`\n",
    "/ `export`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "emix-018",
   "metadata": {},
   "outputs": [],
   "source": [
    "db.to_txt(path=\"/path/to/output\", scenario=\"electricity_mix\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-019",
   "metadata": {},
   "source": [
    "## Excel route\n",
    "\n",
    "Both updates can also be authored in the shock workbook, without writing any\n",
    "code: the coefficient sheets accept the `Supply mix N` and `Trade mix N` types,\n",
    "and `shock_calc(...)` applies them on top of a new scenario. See\n",
    "[Shock analyses](../transformations/apply_shocks.ipynb) for the sheet layouts\n",
    "and worked examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emix-020",
   "metadata": {},
   "source": [
    "## Governed data sources: nxbase and nxsut\n",
    "\n",
    "The live provider APIs (EMBER, ENTSO-E) are one way to feed the mix. A second\n",
    "is to read the same inputs from a governed data backbone that snapshots and\n",
    "versions them, which keeps a pipeline reproducible offline:\n",
    "\n",
    "* **nxbase**, the eNextGen data backbone, exposes EMBER generation and ENTSO-E\n",
    "  trades through a read-only query API. nxbase is currently in **beta** and\n",
    "  public API tokens are not available yet, so this path is not open to external\n",
    "  users at the moment.\n",
    "  [nxbase](https://enextgen.it/en/products/nxbase/)\n",
    "* **nxsut**, the reference supply-use database, is generated with exactly this\n",
    "  pipeline. Its `gen_v3.ipynb` generator parses EXIOBASE Hybrid, updates the\n",
    "  supply mix from EMBER and the trade mix from ENTSO-E (adding sector-specific\n",
    "  detail on top), and is a complete, working end-to-end example.\n",
    "  [nxsut](https://github.com/eNextHub/nxsut)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
