Metadata-Version: 2.4
Name: edgeengine-aware
Version: 0.4.0
Summary: EdgeEngine AWARE: application- and energy-aware simulation environment for reinforcement learning in energy-harvesting Edge IoT systems.
Author-email: Riccardo Berta <riccardo.berta@unige.it>
License-Expression: MIT
Project-URL: Homepage, https://github.com/measurify/EdgeEngine-AWARE
Project-URL: Repository, https://github.com/measurify/EdgeEngine-AWARE
Project-URL: Documentation, https://github.com/measurify/EdgeEngine-AWARE#readme
Project-URL: Issues, https://github.com/measurify/EdgeEngine-AWARE/issues
Keywords: reinforcement-learning,gymnasium,iot,edge,energy-harvesting,tinyml,smart-agriculture,indoor-air-quality,predictive-maintenance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Hardware
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: gymnasium>=0.29
Requires-Dist: matplotlib>=3.7
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: jupyter; extra == "dev"
Requires-Dist: nbformat; extra == "dev"
Requires-Dist: nbconvert; extra == "dev"
Provides-Extra: rl
Requires-Dist: stable-baselines3>=2.0; extra == "rl"
Requires-Dist: sb3-contrib>=2.0; extra == "rl"
Requires-Dist: torch; extra == "rl"
Dynamic: license-file

# EdgeEngine AWARE

[![tests](https://github.com/measurify/EdgeEngine-AWARE/actions/workflows/ci.yml/badge.svg)](https://github.com/measurify/EdgeEngine-AWARE/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/edgeengine-aware.svg)](https://pypi.org/project/edgeengine-aware/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

**Application- and Energy-Aware Simulation Environment for Reinforcement Learning in
Energy-Harvesting Edge IoT Systems**

EdgeEngine AWARE is a Python simulator, packaged as a standard
[Gymnasium](https://gymnasium.farama.org/) environment, of a small **energy-harvesting
embedded node** that monitors a physical quantity and reports it to a remote application over
a low-power radio link. It exists to **train and compare decision policies** — hand-written
rules or reinforcement-learning (RL) agents — that decide, every 15 minutes, how to spend a
very small and uncertain energy budget. Three application domains are built in: a
solar-powered soil-moisture node in a field (the reference case used throughout this README),
a CO₂ node in a classroom powered by the ceiling lights, and a machine-monitoring node powered
by the heat of the motor it watches (§10).

The project also contains what is needed *after* training: a frozen export format for
policies, a firmware-style controller, a C runtime that reproduces the Python decision
logic on a microcontroller, and a way to replay recorded weather data through the same
simulator.

This README is written for a reader who knows Python but not necessarily RL, IoT
hardware or radio engineering. Terms in *italics* on first use are defined in the
[glossary](#glossary) at the end.

---

## Contents

1. [The problem in plain words](#1-the-problem-in-plain-words)
2. [The one design rule](#2-the-one-design-rule-the-policy-must-not-cheat)
3. [Installation](#3-installation)
4. [Quick start](#4-quick-start)
5. [What happens in one step](#5-what-happens-in-one-step)
6. [Observation, action, reward, metrics](#6-observation-action-reward-metrics)
7. [Configuration, scenarios and domain randomisation](#7-configuration-scenarios-and-domain-randomisation)
8. [Built-in policies](#8-built-in-policies)
9. [Training and comparing RL agents](#9-training-and-comparing-rl-agents)
10. [Three application domains](#10-three-application-domains)
11. [Recorded traces instead of models](#11-recorded-traces-instead-of-models)
12. [From simulation to a microcontroller](#12-from-simulation-to-a-microcontroller)
13. [Notebooks, scripts and tools](#13-notebooks-scripts-and-tools)
14. [Repository layout](#14-repository-layout)
15. [Tests](#15-tests)
16. [Documentation](#16-documentation)
17. [What is not modelled](#17-what-is-not-modelled)
18. [Glossary](#glossary)
19. [Citation](#citation)

---

## 1. The problem in plain words

Imagine a matchbox-sized device stuck in the soil of a field:

* It has a **soil-moisture sensor**. Reading it costs energy. A quick, cheap reading is
  noisy; a careful reading (longer warm-up, averaging) is accurate but costs six times more.
* It has a **radio** that can send the latest reading to a gateway several hundred metres
  away. Sending costs energy too, and the radio has three settings: *fast* (cheap, but the
  message often gets lost when the link is poor), *standard*, and *robust* (expensive, almost
  always gets through). Whether a message arrived is known from an acknowledgement (*ACK*).
* It has a **small battery** (300 J, about a 25 mAh cell) charged by a **tiny solar cell**
  that yields about 55 J on an average day and much less on a cloudy one. Just staying alive
  (sleeping, keeping the clock running) costs about 17 J per day.
* A remote **application** (think: the farmer's dashboard) wants to know the soil moisture
  — accurately, and quickly when it matters, i.e. when the soil is getting dry and
  irrigation decisions are due. It tells the node how urgent its needs are (a *priority*
  level: routine, elevated, urgent).

Every 15 minutes the node wakes up and must decide two things: **how well to look** (do not
sense / cheap reading / accurate reading) and **whether and how loudly to speak** (do not
transmit / transmit fast / standard / robust). The score it is judged on is *how right and
how fresh the application's picture of the field is*, minus the energy it spends and minus
penalties for draining the battery. This is a sequential decision problem under uncertainty
— exactly what reinforcement learning is for — but also one where a well-designed set of
rules does very well, which is why the project ships both and compares them honestly.

## 2. The one design rule: the policy must not cheat

A simulator knows everything: the true soil moisture, tomorrow's clouds, whether the next
radio message will get through. A real device knows almost nothing of this. If a policy is
trained on information that the device will not have, it will not work in the field.

EdgeEngine AWARE therefore separates three kinds of quantities and enforces the separation
in code and tests:

| kind | examples | who may use it |
|---|---|---|
| **hidden ground truth** (simulator only) | true soil moisture, true irradiance, radio channel state, future rain | the world dynamics, the reward, the plots |
| **measurable on the node** | stored energy, harvested power in the last interval, the noisy sensor reading and its age, whether the last message was acknowledged, the priority received from the application | the node, i.e. `NodeState` |
| **policy input** | 18 numbers in [0, 1], all computed from the measurable quantities | the policy |

The code that turns measurable quantities into the 18-number *observation*
(`observation.py`), and the code that decides which requested action is affordable
(`actions.plan_execution`), are **shared** between the simulator (`env.py`), the
firmware-style controller (`deployment.NodeController`) and the C runtime (`firmware/`). The
same policy object therefore runs unchanged in all three places. A test
(`tests/test_env.py::test_observation_contains_no_privileged_information`) changes every
hidden variable and checks that the observation does not move.

## 3. Installation

Requirements: Python 3.11 or newer.

**To use the environment** (train or evaluate policies in your own code), install it from
PyPI. The package contains the simulator only — about 100 kB, depending on `numpy`,
`gymnasium` and `matplotlib`:

```bash
pip install edgeengine-aware
pip install "edgeengine-aware[rl]"   # adds stable-baselines3, sb3-contrib, torch (CPU is enough) for RL training
```

**To work with the repository** (notebooks, weather traces, trained policy bundles, the C
runtime, the tests), clone it and install in "editable" mode, so that edits to the source
are picked up without reinstalling. A C compiler (`cc`/`gcc`/`clang`) is optional and only
needed for the firmware equivalence tests.

```bash
git clone https://github.com/measurify/EdgeEngine-AWARE.git
cd EdgeEngine-AWARE
python -m venv .venv && source .venv/bin/activate   # optional but recommended
pip install -e ".[dev]"      # numpy, gymnasium, matplotlib + pytest, jupyter, nbformat, nbconvert
pip install -e ".[rl]"       # only for RL training
pytest                       # 114 tests, 15-60 s depending on the installed extras
```

Everything outside the `edgeengine_aware/` package — `data/traces/` (recorded weather),
`examples/bundles/` (trained policies), `examples/*.ipynb`, `firmware/`, `tools/` — is only
in the repository, not in the pip package. Releases are tagged `vX.Y.Z` on GitHub and
published to PyPI automatically (`.github/workflows/publish.yml`).

## 4. Quick start

**Create the environment and take one step.** `gym.make` returns the environment wrapped in
Gymnasium's standard wrappers; `env.unwrapped` is the underlying `EdgeEngineAwareEnv`.

```python
import gymnasium as gym
import edgeengine_aware as ea            # importing the package registers "EdgeEngineAware-v0"

env = gym.make("EdgeEngineAware-v0")
obs, info = env.reset(seed=0)             # obs: 18 float32 values in [0, 1]
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

print(env.unwrapped.obs_builder.describe())   # what each of the 18 numbers means, from code
print(info["reward_components"])              # utility, energy costs and penalties, separately
print(info["metrics"]["average_aoi_s"])       # running episode metrics (here: mean age of information, s)
```

**Run the rule-based controller for a whole 7-day episode** and print the metrics. The
controller needs the *node profile* (the hardware constants: energy per operation, radio link
budget table) to choose the radio mode; without it, it always uses the standard mode.

```python
import edgeengine_aware as ea
from edgeengine_aware.policies import RuleBasedPolicy, run_episode

cfg = ea.default_config()
env = ea.EdgeEngineAwareEnv(cfg, render_mode="human")       # "human" opens a live Matplotlib dashboard
profile = ea.NodeProfile.from_config(cfg)
result = run_episode(env, RuleBasedPolicy(profile=profile), seed=1, render=True, render_every=96)
print(env.metrics.summary())
```

Typical output (seed 1, default configuration): total reward ≈ 106, 185 transmissions of
which 160 delivered (86 %), minimum battery level 47 %, no brown-out, mean age of
information at the application 0.86 h.

**Compare the baselines without graphics:**

```bash
python examples/compare_policies.py --seeds 10
```

**Open the guided notebook** (environment tour, one episode step by step, metrics, policy
comparison):

```bash
jupyter notebook examples/baseline_policy.ipynb
```

## 5. What happens in one step

One call to `env.step(action)` advances the world by one decision interval (15 minutes by
default) in this order:

1. **Feasibility.** Using only the energy it can read from its battery gauge, the node sets
   aside the always-on consumption of the coming interval and a small brown-out reserve
   (2 % of capacity) and checks which requested operations it can afford. Sensing is served
   first, then transmission. Unaffordable operations are *rejected*: not executed, no energy
   spent, a small penalty. A transmission with nothing to send (no stored reading) is also
   rejected. See `docs/actions.md`.
2. **Sensing.** If requested and affordable, the sensor samples the *true* soil moisture with
   the noise of the chosen level; the reading is stored on the node together with its
   timestamp and a quality tag.
3. **Transmission.** If requested and affordable, the latest stored reading is sent with the
   chosen radio mode. Energy is spent whether or not the message arrives. Delivery is random,
   with a probability that depends on the radio mode and on the current (hidden) state of the
   link. On delivery the application updates its picture of the field; the node learns the
   outcome from the acknowledgement.
4. **Energy update.** Consumption (always-on + sensing + transmission) is drawn from the
   battery first, then the energy harvested during the interval is added, capped at the
   capacity (excess is counted as *wasted*). If the always-on load itself cannot be served
   the node *browns out*.
5. **World update.** Clock, soil, weather, radio channel and application priority move to
   the next interval.
6. **Reward.** How good the application's picture now is (compared with the new truth), plus
   a bonus for the packet that improved it, minus energy costs and penalties.
7. **Observation.** The node's bookkeeping is refreshed with measurable quantities only and
   the 18-number vector for the next decision is built.

The full timeline, with the formulas, is in `docs/modeling.md`.

## 6. Observation, action, reward, metrics

**Observation** — `Box(0, 1, (18,), float32)`. Every component is normalised to [0, 1] and
comes from something a microcontroller can measure or compute:

| group | components |
|---|---|
| energy | battery *SoC*; harvested power in the last interval; smoothed recent harvest (*EWMA*) |
| time | time of day as sine and cosine |
| stored reading | value; quality tag (0 none / 0.2 cheap / 0.8 accurate); age |
| communication | time since the last acknowledged uplink; the node's own estimate of how old the application's information is; the value it last delivered; smoothed fraction of recent uplinks acknowledged (*link quality*); *path-loss* estimate from the last ACK |
| application | requested priority (0 / 0.5 / 1) |
| node-side judgement | *importance* of the stored reading (how close it is to the stress thresholds) |
| hardware profile | energy of a cheap reading, of an accurate reading, of one standard uplink, all relative to the battery capacity |

Full table with normalisation formulas and hardware sources: `docs/observation.md`.

**Action** — `MultiDiscrete([3, 4])`, i.e. a pair of integers:

| component | values |
|---|---|
| `sensing_level` | 0 = do not sense, 1 = cheap reading (0.10 J, noise σ 0.04), 2 = accurate reading (0.60 J, σ 0.01) |
| `transmit` | 0 = do not transmit, 1 = fast mode (0.30 J), 2 = standard (0.60 J), 3 = robust (1.20 J) |

For algorithms that need a single integer (DQN, tabular methods) `rl.FlatActionWrapper`
exposes the same choices as `Discrete(12)` (`12 = 3 × 4`; it grows if you configure more
radio modes). See `docs/actions.md`.

**Reward** — a sum of separately reported terms, available in `info["reward_components"]`:

| term | sign | meaning |
|---|---|---|
| tracking utility | + | every step: how close the application's last value is to the true moisture, weighted up to 3× near the stress thresholds |
| packet bonus | + | on delivery: signed improvement of the application's error, plus a bonus for reporting a recent rain/irrigation event |
| energy costs | − | 0.1 per joule spent on sensing and on transmission |
| staleness | − | grows with the age of the application's information, weighted 1/2/4 by priority, saturating after 6 h |
| battery risk | − | quadratic penalty below 30 % SoC; 2.0 per step of brown-out |
| rejected actions, wasted harvest | − | 0.2 per rejected operation; 0.02 per joule of harvest that did not fit in the battery |

Orders of magnitude for a 7-day episode with the defaults: utility 100–140, energy costs
15–30, a policy that never reports loses ~130 to staleness, an always-on policy loses several
hundred to battery penalties. Formulas and constants: `docs/modeling.md` §3.

**Metrics** — `env.metrics` (an `EpisodeMetrics`) accumulates per episode: harvested /
consumed / always-on / sensing / communication / wasted energy [J]; number of readings,
accurate readings, transmissions, deliveries (also per radio mode), rejected operations;
average and minimum SoC, fraction of steps below 30 % SoC, brown-out events; average and
maximum *age of information* (AoI); total utility and reward. `env.metrics.summary()` prints
them, `info["metrics"]` carries the same dictionary at every step, and `env.log` (an
`EpisodeLog`) keeps the per-step history used by the plots.

**Other `info` keys.** `ground_truth` (hidden state, for evaluation only — never feed it to
a policy), `rejected` (list of reasons), `tx_mode`, `tx_success`, `app_priority` (what the
application wants now) vs `node_priority` (what the node has been told), `utility_breakdown`,
`observation_names`.

## 7. Configuration, scenarios and domain randomisation

Everything is a plain Python dataclass; the environment code never needs to change.

```python
import edgeengine_aware as ea

cfg = ea.default_config()
cfg.time.timestep_s = 600                     # 10-minute decisions
cfg.time.episode_days = 14
cfg.storage.capacity_j = 800
cfg.storage.initial_soc_range = (0.3, 0.9)    # random initial charge at every reset
cfg.mcu.baseline_power_w = 120e-6
cfg.harvesting.max_power_w = 0.008            # panel peak power [W] at clear-sky noon
cfg.observation.harvest_ref_power_w = 0.005   # keep the harvest normalisation consistent with the panel
cfg.harvesting.clearness_mean = 0.5           # cloudier climate
cfg.communication = ea.config.single_mode_radio(energy_j=0.9)   # a radio with one setting -> binary transmit action
cfg.communication.priority_update_mode = "on_uplink"   # priority only arrives with an ACK (LoRaWAN class A)
env = ea.EdgeEngineAwareEnv(cfg)
```

Two things to know: `cfg.validate()` is called by the environment and rejects inconsistent
settings with a clear message; and the observation normalisation constants
(`cfg.observation`) are part of the hardware profile — if you change the panel or the
battery, check that `harvest_ref_power_w` (the power that maps to 1.0) still makes sense.

**Scenarios** (`edgeengine_aware.scenarios`) are named configurations used as the benchmark
protocol. Each stresses one aspect:

| name | what changes with respect to `default` | what it tests |
|---|---|---|
| `default` | — (mixed weather, clearness 0.7 ± 0.2) | the nominal regime |
| `cloudy_week` | clearness 0.35 ± 0.15, more cloud variability, battery starts at 40 % → ~30 J/day (55 % of default) | rationing energy over several dark days |
| `tiny_battery` | 150 J instead of 300 J | smoothing consumption with a small buffer |
| `lossy_link` | +6 dB mean path loss, slower fading → standard mode delivers ~40 %, robust ~85 % | choosing the radio mode from the link estimate |
| `drought` | evapotranspiration ×2, almost no rain, irrigation delayed 18 h, drier start | tracking the approach to the stress thresholds |
| `demanding_application` | 1.5 external campaigns/day of 2–8 h, half of them urgent, "elevated" after 4 h of silence | following the priority, saving in between |

`ea.get_scenario(name, randomize=False)` returns the configuration; `rl.make_env(name)` the
environment; `rl.make_env("mixed")` (or a list of names) an environment that draws a scenario
at every reset (`MixedScenarioEnv`) — the recommended training distribution.

**Domain randomisation** (`cfg.randomization`, off by default) perturbs the physical
parameters at every `reset()` so that a trained policy does not over-fit one exact device:
sensor noise ×[0.7, 1.5], sensing energy ×[0.8, 1.3], radio energies ×[0.8, 1.3], mean path
loss ±3 dB, solar intensity ×[0.6, 1.2], cloud variability ×[0.5, 1.5], battery capacity
×[0.8, 1.2], always-on power ×[0.7, 1.5] (all uniform); in the domains with a weekly schedule
(§10) the weekday on which the episode starts is drawn at random too. The randomised energy costs are what
the node reports in its observation, exactly as a real device would report its own measured
profile. Enable with `cfg.randomization.enabled = True` or `get_scenario(name, randomize=True)`.

## 8. Built-in policies

All policies implement the tiny `Policy` protocol — `act(observation) -> action` and
`reset()` — and only ever see the 18-number observation, so they run identically in the
simulator, in the deployment controller and (for the rule-based one) in C.

| policy | what it does |
|---|---|
| `RuleBasedPolicy(params, profile)` | the interpretable reference controller, see below |
| `PeriodicPolicy(period_steps, sensing_level, tx)` | the classic duty-cycled firmware: sense and transmit every *n* steps, regardless of energy or application |
| `RandomPolicy(seed)` | uniform random actions — a lower bound |
| `AlwaysOnPolicy()` | accurate reading and transmission at every step — an upper bound on information, a disaster for the battery |

**The rule-based controller** (`policies.RuleBasedPolicy`, parameters in `RuleBasedParams`)
applies these rules in order, all on the normalised observation:

1. *Deep economy* — battery below 20 %: one accurate report every 8 h (every 30 min if the
   application is urgent), nothing else.
2. *Retry* — a fresh accurate reading (younger than 18 min) that was not acknowledged is
   re-sent, unless the link looks down (link quality < 0.5).
3. *Scheduled report* — when the estimated age of the application's information exceeds the
   report interval: 2 h / 1 h / 30 min for routine / elevated / urgent priority. In *economy
   mode* (battery below 50 %, unless the priority is urgent) the interval is doubled; at
   routine priority it is shortened to 75 % when the battery is above 70 % or the sun is
   strong. A report is an accurate reading plus a transmission.
4. *Event report* — the stored reading differs from the last delivered one by more than 0.08,
   or is near a stress threshold and differs by more than 0.03.
5. *Check* — outside economy mode, a cheap reading every hour (no transmission).
6. Otherwise sleep.

Radio mode: the cheapest mode whose expected link margin, computed from the node's path-loss
estimate and the link-budget table in the profile, is at least 4 dB; one mode more robust
when recent uplinks failed (link quality < 0.7); the standard mode before the first estimate.

## 9. Training and comparing RL agents

The environment follows the Gymnasium API, so any library works. The project uses
[Stable-Baselines3](https://stable-baselines3.readthedocs.io/) (SB3): PPO and A2C accept the
`MultiDiscrete` action space directly; DQN needs `rl.FlatActionWrapper`. No RL algorithm is
implemented inside the package on purpose — the package is the benchmark, the notebook is
the experiment.

```python
import edgeengine_aware as ea
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv
from edgeengine_aware.rl import make_env_fn, SB3Policy, evaluate, summarize
from edgeengine_aware.scenarios import SCENARIOS

# train on a mixture of all scenarios with domain randomisation, 8 parallel copies
vec_env = DummyVecEnv([make_env_fn("mixed", randomize=True, seed=i) for i in range(8)])
model = PPO("MlpPolicy", vec_env, policy_kwargs=dict(net_arch=[64, 64]), device="cpu").learn(2_000_000)

# evaluate on every scenario, 20 held-out seeds, against the rule-based controller
profile = ea.NodeProfile.from_config(ea.default_config())
rows = evaluate(
    {"PPO": lambda: SB3Policy(model), "rule-based": lambda: ea.RuleBasedPolicy(profile=profile)},
    scenarios=SCENARIOS, seeds=range(1000, 1020),
)
print(summarize(rows, "reward"))     # {scenario: {policy: (mean, std)}}
```

**Evaluation protocol** (`rl.evaluate`): deterministic policies, 7-day episodes, the six
scenarios with nominal (non-randomised) physics, seeds ≥ 1000 that training never uses; one
`EvalRow` per episode with reward, utility, energy, counts, min SoC, brown-outs, AoI and the
reward components. `evaluate(..., envs={label: env})` evaluates on ready-made environments
instead (e.g. trace-driven ones, §11, or other domains, §10).

**What we found** (`examples/train_rl.ipynb`, details in the notebook): with 1–2 M training
steps PPO learns a sensible policy — including using the robust radio mode when the link is
poor and never the fast one — but the link-aware rule-based controller is as good or better
on the nominal, lossy-link and small-battery scenarios; PPO wins on the energy-limited
`cloudy_week`. With **5 M steps** (three seeds, `examples/run_long_training.sh`) PPO reaches
parity with the rule-based controller on the nominal, small-battery, lossy-link and
demanding-application scenarios and is clearly ahead on `cloudy_week` (+6) and `drought`
(+8): 82.0 ± 1.8 vs 81.1 averaged over the six scenarios. DQN is clearly below PPO at every
budget tried, with a higher variance across seeds, and wastes uplinks in the fast mode. Adding
memory (4-frame stacking or a recurrent PPO) does not help. Repeating training over several
seeds (`examples/train_seeds.py`) gives the mean ± std that a paper should quote.

## 10. Three application domains

The agricultural node is one instance of a general problem: an embedded node with a small
energy store, a harvester and a duty to report. `edgeengine_aware.domains` makes the hidden
world pluggable and ships three domains. Whatever the domain, the node sees the **same
18-number observation** and returns the **same action**: the monitored quantity is normalised
to [0, 1], and the only new profile constant is on which side the danger lies
(`critical_is_upper`). The rule-based controller, the RL tooling, the exported bundles and the
C runtime therefore work unchanged in every domain — and a policy trained in one domain can be
evaluated in another.

| domain | monitored quantity (danger) | energy source | radio | storage / always-on | what drives the week |
|---|---|---|---|---|---|
| `agriculture` | soil moisture (low) | solar cell, weather | LoRa-like, 3 spreading factors | 300 J / 200 µW | sun, clouds, rain, irrigation |
| `indoor_air` | CO₂ of a classroom (high: 1000 / 1500 ppm) | indoor PV under the ceiling lights, plus a window | BLE-like, 3 PHY modes, sub-millijoule uplinks | 60 J / 20 µW | occupancy 08–18 on weekdays: people raise the CO₂ *and* switch the lights on; nights and weekends bring neither energy nor relevance; the CO₂ sensor (20 / 150 mJ per reading) is the energy hog |
| `industrial` | bearing temperature of a motor (high: 70 / 90 °C) | thermoelectric generator on the warm casing | LoRa-like | 120 J / 200 µW | two shifts 06–22 on weekdays: the machine's heat is both the energy source and the monitored variable; faults make it run hotter until maintenance; weekends are cold and dark |

```python
import edgeengine_aware as ea
from edgeengine_aware.rl import make_env

env = ea.EdgeEngineAwareEnv(ea.domain_config("industrial"))      # a domain's default configuration
env = make_env("indoor_air:no_window")                          # a scenario of another domain ("domain:scenario")
env = make_env("all")                                           # training mixture over every scenario of every domain
print(ea.scenario_names("all"))                                 # 6 agricultural + 5 + 5 scenarios
```

Each domain has its own stress scenarios (`indoor_air`: `no_window`, `weak_ventilation`,
`long_hours`, `dim_lights`; `industrial`: `single_shift`, `degrading`, `continuous`,
`weak_link`). `examples/domains.ipynb` puts the three side by side and asks the cross-domain
question. First answer: the 5 M-step PPO trained on the agricultural scenarios is at parity
with the rule-based controller in its own domain but **loses 21 points on the industrial and
33 on the indoor domain** — it learned the solar day, not only the trade-off.
`examples/run_cross_domain.sh` trains one policy per domain and a *universal* one on the
mixture of all three (PPO, 5 M steps, two seeds); the notebook builds the train-domain ×
test-domain matrix from those runs. Advantage over the rule-based controller, averaged over the
test domain's scenarios:

| trained on ↓ / tested on → | agriculture | indoor_air | industrial |
|---|---|---|---|
| agriculture | **+4** | −24 | −14 |
| indoor_air | −668 | **+51** | −526 |
| industrial | −9 | +13 | **−3** |
| all three (universal) | −6 | +46 | −11 |

Three things to read in it. There is **no transfer without training** — the indoor policy,
which learned that uplinks are free (1 mJ over BLE), drains a LoRa node's battery in a day.
**In its own domain PPO is at parity with the rule on the two physical domains and far ahead
on the indoor one** (+51; a rule re-tuned for the domain recovers only part of it), where
relevance is bursty and uplinks cheap, so the learned policy reports every 15 minutes while
the room is occupied and sleeps otherwise. **The universal policy** keeps most of the indoor
gain and stays within ~10 points of the rule elsewhere: one network can serve three products
at a modest cost, with two seeds and 5 M steps. The exported policies are shipped in
`examples/bundles/` (`ppo_indoor_air.json`, `ppo_industrial.json`, `ppo_universal.json`).
Models and constants of the two new domains: `docs/modeling.md` §6.

## 11. Recorded traces instead of models

The weather and soil dynamics of the simulator are *stochastic models*. To check a policy
against data those models never produced, `edgeengine_aware.traces` replays recorded hourly
series through the same environment:

* `Trace` — a time-indexed table loaded from CSV. Each column is either *instant* (soil
  moisture, temperature: linearly interpolated) or a *preceding-interval mean* (irradiance
  in W/m², precipitation in mm — the convention of weather archives, held constant over the
  hour that ends at the timestamp). Time is in seconds since the first local midnight.
* `TraceSolarEnergySource` — panel power = `max_power_w × efficiency × irradiance / 1000 W/m²`.
* `TraceFieldEnvironment` — replays soil moisture (volumetric m³/m³ divided by the field
  capacity, 0.40 by default, to get the simulator's "fraction of field capacity"),
  temperature, humidity and rain as the hidden ground truth.
* `TraceDrivenEnv(trace, config, start_day=None, harvesting=True, field=True, ...)` — an
  `EdgeEngineAwareEnv` whose harvesting and/or field come from the trace. Each `reset()`
  replays a 7-day window starting at a random admissible day (or at `start_day`, or at
  `reset(options={"start_day": k})`); `info["trace_start_day"]` reports it. Spaces,
  observation and reward are unchanged: **the policy cannot tell the two worlds apart.**

```python
import edgeengine_aware as ea
from edgeengine_aware.traces import Trace, TraceDrivenEnv

trace = Trace.from_csv("data/traces/demo_liguria_2023_hourly.csv")
print(trace.describe())
summer = trace.slice_days(151, 92)                     # days 151-243 of the year
env = TraceDrivenEnv(summer, ea.default_config())      # random summer weeks at every reset
obs, info = env.reset(seed=0)
```

**Data.** `data/traces/albenga_2023_hourly.csv` is the real ERA5 / ERA5-Land reanalysis for
a coastal site in Liguria (year 2023, hourly), downloaded with `tools/fetch_open_meteo.py`
(Open-Meteo archive, licence CC BY 4.0; the script needs only the standard library and works
for any site and year). `data/traces/demo_liguria_2023_hourly.csv` is a clearly labelled
**synthetic** stand-in with a darker winter, generated by `tools/make_demo_trace.py`. Your own
logger data works with the same CSV format (`data/traces/README.md`).
`examples/trace_driven.ipynb` compares the trace statistics with the synthetic scenarios and
evaluates the baselines and the exported PPO policies season by season: on the real trace the
PPO policies trained on the synthetic mixture behave as on the synthetic scenarios — the
2 M-step policy about 3 points below the rule-based controller, the 5 M-step policy at parity
while sending fewer packets — with no brown-outs in winter; on the darker synthetic trace the
2 M policy fails in winter, the regime the training scenarios do not contain.

## 12. From simulation to a microcontroller

Three pieces make a trained policy deployable:

1. **`deployment.PolicyBundle`** — one JSON file that freezes everything the device must
   reproduce: the observation names in order and their normalisation formulas, the node
   profile (all constants), the action encoding, the model parameters (thresholds for the
   rule-based policy, weights for a neural network) and metadata. `export_policy(...)` creates
   it; `rl.export_sb3_mlp(model)` extracts the weights of an SB3 network as plain lists.
   Six bundles are shipped in `examples/bundles/`: `rule_based_default.json`,
   `ppo_default.json` (PPO, 2 M steps, network 18→64→64→7), `ppo_long.json` (PPO, 5 M steps,
   agriculture), `ppo_indoor_air.json`, `ppo_industrial.json` and `ppo_universal.json` (PPO,
   5 M steps, trained on that domain or on all three).
2. **`deployment.NodeController`** — the firmware main loop written in Python against the
   hardware protocols of `interfaces.py` (clock, energy storage, energy source, sensor,
   radio, application). It reuses the simulator's tracker, observation builder and
   feasibility rule. `deployment.make_mock_backend(profile)` provides fake drivers with no
   ground truth so the loop can be exercised on a laptop; `tests/test_deployment.py` replays
   a simulated episode's driver readings through it and checks that the observations match
   the simulator's bit for bit.
3. **`firmware/`** — a dependency-free C99 translation of the node-side logic: tracker,
   observation builder, feasibility rule, rule-based controller and the dense-network forward
   pass (about 4 kB of code). `tools/export_c.py` turns a bundle into a header with the
   constants and weights; `firmware/main_example.c` is the decision loop with stub drivers to
   replace; `tests/test_firmware.py` compiles it and checks against Python: observations
   **bit-exact** (float32), rule-based and feasibility decisions identical, network outputs
   equal to ~1e-6 and actions identical except at exact numerical ties.

```bash
python tools/export_c.py examples/bundles/ppo_default.json -o firmware/eea_policy_data.h
cc -std=c99 -O2 -I firmware firmware/eea_node.c firmware/main_example.c -lm -o node_example && ./node_example
```

Details, the field-evaluation workflow and the alternatives (TensorFlow Lite Micro, CMSIS-NN):
`docs/deployment.md`, `firmware/README.md`.

## 13. Notebooks, scripts and tools

The notebooks are **generated** by the `examples/build_*.py` scripts (edit the script, not the
notebook, then run it and execute the notebook).

| file | purpose | runtime |
|---|---|---|
| `examples/baseline_policy.ipynb` | guided tour: environment, one episode step by step, dashboard, metrics, baseline comparison over scenarios | < 1 min |
| `examples/train_rl.ipynb` | the RL experiment: protocol, baselines, PPO and DQN training, learning curves, comparison, behaviour analysis, export check, multi-seed study, memory study | `EEA_BUDGET=quick` ~3 min (smoke test), `default` ~30–40 min + ~1.5 h for the seed study, `paper` longer |
| `examples/trace_driven.ipynb` | the simulator on recorded traces: seasons, model-vs-trace statistics, policies per season, replayed weeks | ~1 min |
| `examples/domains.ipynb` | the three domains side by side: energy budgets, one week each, baselines on all 16 scenarios, cross-domain transfer of PPO policies | ~3 min |
| `examples/compare_policies.py` | headless baseline comparison (`--seeds`, `--randomize`, `--days`) | seconds |
| `examples/train_seeds.py` | train and evaluate one (algorithm, seed) pair: `--algo ppo|dqn|ppo_stack|rppo --seed N --steps ... --domain agriculture|indoor_air|industrial|all --eval-domain ... --out DIR`; writes a JSON with the evaluation rows and learning curve, the checkpoints and (PPO/DQN) the exported bundle | 20–60 min per run |
| `examples/run_long_training.sh` | chains the long runs (PPO 5 M steps × 3 seeds, DQN 2 M × 3) into `examples/rl_runs/seeds_long/`, creating `.venv` with the RL extras if needed; resumable, keeps a macOS laptop awake | 1–5 h |
| `examples/run_cross_domain.sh` | PPO 5 M × 2 seeds trained on each domain and on the mixture of all, evaluated on every domain → `examples/rl_runs/seeds_domains/` (read by `domains.ipynb` §4b) | ~2 h |
| `tools/fetch_open_meteo.py` | download an hourly ERA5/ERA5-Land trace (`--lat --lon --start --end --utc-offset --site --out`) | seconds |
| `tools/make_demo_trace.py` | regenerate the synthetic demo trace (`--year --seed --out`) | seconds |
| `tools/export_c.py` | `PolicyBundle` JSON → `eea_policy_data.h` (`-o`) | instant |

Trained models and evaluation caches go to `examples/rl_runs/` (ignored by git; set
`EEA_RETRAIN=1` to force retraining).

## 14. Repository layout

```
edgeengine_aware/           the package
  config.py                 every parameter as a dataclass field, with units, defaults and validation; scenarios build on it
  interfaces.py             the six hardware protocols (Clock, EnergyStorage, EnergySource, Sensor, Radio, RemoteApplication),
                            the Measurement / Packet / TxResult records and the Policy protocol
  observation.py            NodeProfile (constants), NodeState (measurable state), NodeStateTracker (bookkeeping),
                            ObservationBuilder (the 18-vector) - the sim-to-real contract
  actions.py                action encoding, flat index, plan_execution (feasibility rule)
  energy.py                 SimulatedClock, SimulatedEnergyStorage, SolarEnergySource (stochastic solar model)
  agriculture.py            FieldEnvironment: hidden soil / weather ground truth of the agriculture domain
  sensing.py                SimulatedSoilMoistureSensor
  communication.py          SimulatedLoRaRadio: radio modes, link-budget channel with slow and fast fading, ACK + margin
  application.py            RemoteMonitoringApplication: utility of the delivered information, priority requests
  reward.py                 RewardCalculator with separately reported components
  metrics.py                EpisodeMetrics and EpisodeLog
  env.py                    EdgeEngineAwareEnv (gymnasium.Env)
  rendering.py              Matplotlib dashboard (human / rgb_array) and text dashboard (ansi)
  policies.py               RuleBasedPolicy, PeriodicPolicy, RandomPolicy, AlwaysOnPolicy, run_episode
  scenarios.py              the six agricultural benchmark scenarios; lookup of every domain's scenarios by "domain:name"
  domains.py                the three domains: default configurations, their scenarios, the world factory (build_world)
  process.py                ProcessState (what every monitored process reports) and the weekly ActivitySchedule
  indoor.py                 indoor_air domain: CO2 mass balance of a room, indoor-light PV source
  industrial.py             industrial domain: bearing thermal model with wear/faults/maintenance, thermoelectric source
  rl.py                     FlatActionWrapper, MixedScenarioEnv, make_env, SB3Policy, evaluate / summarize,
                            export_sb3_mlp, NumpyMLPPolicy, FrameStacker / StackedPolicy
  traces.py                 Trace, TraceSolarEnergySource, TraceFieldEnvironment, TraceDrivenEnv
  deployment.py             NodeController, mock hardware backend, PolicyBundle, export_policy
firmware/                   C99 runtime (eea_node.h/.c), main_example.c, test harness, README
tools/                      export_c.py, fetch_open_meteo.py, make_demo_trace.py
data/traces/                real Albenga 2023 trace (ERA5-Land), synthetic demo trace, CSV format description
examples/                   notebooks, their generator scripts, compare_policies.py, train_seeds.py, run_long_training.sh, bundles/
tests/                      pytest suite (see below)
docs/                       detailed documentation (see below)
```

## 15. Tests

`pytest` runs 114 tests:

| file | what it checks |
|---|---|
| `test_env.py` | Gymnasium API and spaces, `check_env`, truncation, feasibility/rejection, no leakage of hidden state into the observation, rendering, single-mode radio |
| `test_energy.py` | energy bounds and conservation, units, solar model, storage bookkeeping |
| `test_reproducibility.py` | same seed → same episode; domain randomisation changes the physics but not the spaces |
| `test_deployment.py` | protocols, mock backend, controller vs simulator observation equality, bundle export |
| `test_rl.py` | scenarios, wrappers, evaluation protocol, numpy MLP runtime vs SB3, rule-based radio-mode choice |
| `test_traces.py` | trace loader semantics (interpolation, interval means, CSV round trip), trace-driven backends and environment |
| `test_firmware.py` | C runtime vs Python for the three domain profiles (compiles `firmware/` with the system C compiler; skipped if none) |
| `test_domains.py` | threshold direction, weekly schedule, CO₂ and bearing models, TEG coupling, shared spaces across domains, qualified scenario names and mixtures, agriculture defaults unchanged |

## 16. Documentation

| document | content |
|---|---|
| `docs/observation.md` | the 18 observation components: meaning, normalisation, hardware source; why the problem is partially observable and what is done about it |
| `docs/actions.md` | action encoding, radio modes and their link budget, the feasibility rule, flat encoding |
| `docs/modeling.md` | all physical models with formulas and default constants, utility and reward, the MDP/POMDP formulation |
| `docs/sim_to_real.md` | the architecture that keeps simulation and hardware on the same contract, the known gaps and their mitigations, domain randomisation, how to add backends |
| `docs/deployment.md` | PolicyBundle, the controller loop, the C runtime, TinyML alternatives, field-evaluation workflow |
| `firmware/README.md` | the C runtime: API, generated data header, equivalence tests, porting notes |
| `data/traces/README.md` | the trace CSV format and the shipped/obtainable data |

## 17. What is not modelled

Being explicit about the simplifications matters more than the list of features:

* **Battery**: an ideal energy buffer with a single charge-efficiency knob; no voltage
  curve, temperature effect, ageing or self-discharge. The node reads its energy exactly.
* **Radio**: delivery depends only on the link margin; no duty-cycle limits, collisions,
  gateway congestion or multiple gateways; energy per uplink is constant per mode.
* **Soil**: a single bucket with evapotranspiration, rain and external irrigation; no crop
  model, no spatial variability. Irrigation is *not* a decision of the node.
* **Application**: a rule-based priority (thresholds, information age, random campaigns), not
  a human operator.
* **Clock**: exact 15-minute steps, no drift or jitter.
* **Indoor air**: a single well-mixed room with one occupancy schedule; the CO₂ sensor's own
  warm-up and drift are folded into the two energy/noise levels. **Industrial**: a first-order
  thermal model with a scalar "health"; vibration is not simulated, the accurate sensing level
  stands for a vibration burst only through its energy and precision.

`docs/sim_to_real.md` lists, for each of these, what a real device will do differently and
which mitigation (domain randomisation, configuration, recorded traces) is available.

## Glossary

* **ACK (acknowledgement)** — a short reply from the gateway confirming that an uplink was
  received. It also lets the node measure how much margin the link had.
* **AoI (age of information)** — how old the freshest value known to the application is:
  time since it was received plus the age the reading already had when sent.
* **Brown-out** — the battery cannot even supply the always-on load; the node stops
  working until energy is harvested again.
* **Domain randomisation** — perturbing the simulator's physical parameters at every
  episode so that a trained policy is robust to devices and conditions that differ from the
  nominal ones.
* **Duty cycle** — a fixed schedule ("sense and transmit every hour"), the traditional way
  low-power nodes are programmed.
* **EWMA (exponentially weighted moving average)** — a running average that gives weight
  α to the newest value and (1−α) to the previous average; a cheap memory of the recent past.
* **Field capacity** — the amount of water a soil holds after excess water has drained;
  soil moisture in this project is expressed as a fraction of it (1.0 = field capacity).
* **Held-out seeds** — random seeds used only for evaluation, never during training, so that
  results are not flattered by memorised episodes.
* **Link margin / link budget** — transmit power minus path loss minus receiver sensitivity,
  in dB. Positive margin: the message is likely to be received. Each radio mode has a
  different sensitivity, hence a different margin for the same channel.
* **Path loss** — attenuation of the radio signal between node and gateway, in dB; it varies
  slowly (vegetation, humidity, obstacles: *slow fading*) and from message to message
  (*fast fading*).
* **POMDP** — a decision problem in which the agent does not observe the full state (here:
  the true soil moisture, the weather regime, the channel state are hidden). Standard remedies
  are summary statistics of the past, explicit timers and, if needed, memory in the policy.
* **Priority** — the application's declared level of interest: routine (0), elevated (1),
  urgent (2).
* **SF (spreading factor)** — a LoRa modulation setting; higher SF means longer, slower
  transmissions that can be received at weaker signal (better sensitivity) but cost more
  energy. The project's *fast / standard / robust* modes are SF7 / SF9 / SF12-like.
* **SoC (state of charge)** — stored energy divided by capacity, from 0 to 1.
* **`MultiDiscrete([3, 4])`** — Gymnasium's action space for a vector of independent
  integers, here one in {0,1,2} and one in {0,1,2,3}.

## Citation

If you use EdgeEngine AWARE in academic work, please cite it. The repository carries a
`CITATION.cff` file (GitHub shows it under *Cite this repository*); in BibTeX:

```bibtex
@software{elios_edgeengine_aware_2026,
  author  = {{ELIOS Lab, DITEN, University of Genoa}},
  title   = {EdgeEngine AWARE: an application- and energy-aware simulation environment for
             reinforcement learning in energy-harvesting Edge IoT systems},
  year    = {2026},
  version = {0.4.0},
  url     = {https://github.com/measurify/EdgeEngine-AWARE},
  note    = {\url{https://www.elios.unige.it/}}
}
```

EdgeEngine AWARE is released under the MIT License (see `LICENSE`).
