Metadata-Version: 2.4
Name: mrg-iot
Version: 1.4.0
Summary: IoT Testbed Experiment Automation CLI for MergeTB / Sphere Testbed
Author: Sphere / NEU IoT Testbed contributors
License: MIT
Project-URL: Homepage, https://gitlab.com/sphere-neu/mrg-iot
Project-URL: Repository, https://gitlab.com/sphere-neu/mrg-iot
Project-URL: Issues, https://gitlab.com/sphere-neu/mrg-iot/-/issues
Keywords: iot,testbed,mergetb,sphere,automation,cli
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: bcrypt>=5.0.0
Requires-Dist: betterproto==2.0.0b7
Requires-Dist: certifi
Requires-Dist: cffi>=2.0.0
Requires-Dist: cryptography>=46.0.3
Requires-Dist: grpclib>=0.4.8
Requires-Dist: h2>=4.3.0
Requires-Dist: hpack>=4.1.0
Requires-Dist: hyperframe>=6.1.0
Requires-Dist: mergetbapi==1.3.41
Requires-Dist: multidict>=6.7.0
Requires-Dist: packaging>=25.0
Requires-Dist: paramiko>=4.0.0
Requires-Dist: pycparser>=2.23
Requires-Dist: PyNaCl>=1.6.0
Requires-Dist: python-dateutil>=2.9.0
Requires-Dist: requests>=2.32.0
Requires-Dist: rich>=13.0.0
Requires-Dist: six>=1.17.0
Requires-Dist: typing_extensions>=4.15.0
Provides-Extra: dev
Requires-Dist: astroid==4.0.1; extra == "dev"
Requires-Dist: black==25.9.0; extra == "dev"
Requires-Dist: dill==0.4.0; extra == "dev"
Requires-Dist: invoke==2.2.1; extra == "dev"
Requires-Dist: isort==7.0.0; extra == "dev"
Requires-Dist: mccabe==0.7.0; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: mypy_extensions==1.1.0; extra == "dev"
Requires-Dist: pathspec==1.1.1; extra == "dev"
Requires-Dist: platformdirs==4.5.0; extra == "dev"
Requires-Dist: pylint==4.0.2; extra == "dev"
Requires-Dist: pytokens==0.2.0; extra == "dev"
Requires-Dist: tomlkit==0.13.3; extra == "dev"
Requires-Dist: types-paramiko; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Dynamic: license-file

# mrg-iot

> IoT testbed experiment automation for the [Sphere / Merge Testbed](https://sphere-testbed.net) platform.

[![PyPI version](https://img.shields.io/pypi/v/mrg-iot.svg)](https://pypi.org/project/mrg-iot/)
[![Python versions](https://img.shields.io/pypi/pyversions/mrg-iot.svg)](https://pypi.org/project/mrg-iot/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

`mrg-iot` is a command-line tool that drives the **entire lifecycle** of an IoT
experiment on the Merge Testbed — so you don't have to stitch together a dozen
`mergetb` and `ssh` calls by hand. From a single command it will:

1. **Create** an experiment and generate its network model for the devices you select.
2. **Realize** the model (allocate physical hardware).
3. **Materialize** the realization (bring the allocation online).
4. **Create an XDC** (eXperiment Development Container) and **attach** it to the materialization.
5. **SSH** into the XDC and open **SSH tunnels** for RTSP video streams (`:8554`), the SFTPGo file client used for uploads and downloads (`:9001`), and the experiment's `ExperimentControl` gRPC channel (`:17000`).
6. Give you an **interactive control loop** to drive devices, watch camera feeds in VLC, and pull experiment artifacts.
7. **Clean up** every resource it created — even on partial failure.

---

## Table of contents

- [Requirements](#requirements)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Authentication](#authentication)
- [How sessions work](#how-sessions-work)
  - [Two required arguments, everything else derived](#two-required-arguments-everything-else-derived)
  - [One command for the whole lifecycle: `run`](#one-command-for-the-whole-lifecycle-run)
  - [Step by step](#step-by-step)
  - [Commands that take time: `run`, `async-run`, and tasks](#commands-that-take-time-run-async-run-and-tasks)
  - [From inside the XDC: `cmd ctl`](#from-inside-the-xdc-cmd-ctl)
  - [Older spellings](#older-spellings)
- [Files](#files)
- [The `spiot_ctl` control language](#the-spiot_ctl-control-language)
- [Command reference](#command-reference)
- [Input validation rules](#input-validation-rules)
- [Configuration & on-disk state](#configuration--on-disk-state)
- [Architecture](#architecture)
- [Batch device traffic testing](#batch-device-traffic-testing)
- [Troubleshooting](#troubleshooting)
- [Development](#development)
- [Release process](#release-process)
- [License](#license)

---

## Requirements

| Requirement | Notes |
|-------------|-------|
| **Python 3.9+** | 3.9, 3.10, 3.11, and 3.12 are supported and CI-tested. |
| **A Merge Testbed account** | With access to at least one IoT project (`neuiot`, `iotbeta`, or `iotdev`). |
| **Network access** | To the gRPC portal (`grpc.sphere-testbed.net:443`) and the SSH jump host (`jump.sphere-testbed.net:2022`). |
| **[VLC](https://www.videolan.org/vlc/)** | **Only** needed for `stream open` / RTSP playback. `pip` cannot install it — install it from your OS package manager. The `vlc` binary must be on your `PATH`. |

---

## Installation

The supported, production install path is [`pipx`](https://pipx.pypa.io/) — it
puts `mrg-iot` on your `PATH` in its own isolated environment:

```bash
pipx install mrg-iot
```

Or, into a regular virtual environment:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install mrg-iot
```

Verify the install:

```bash
mrg-iot --version
mrg-iot --help
```

### Installing VLC (for video)

| Platform | Command |
|----------|---------|
| macOS | `brew install --cask vlc` |
| Debian/Ubuntu | `sudo apt-get install vlc` |
| Fedora | `sudo dnf install vlc` |
| Windows | [Download from videolan.org](https://www.videolan.org/vlc/) |

---

## Quick start

```bash
# 1. Authenticate once — credentials are cached under ~/.mrg-iot/session.json
mrg-iot login

# 2. Spin up an experiment end-to-end, interactively
mrg-iot run

# ...or fully scripted, no prompts:
mrg-iot run --non-interactive \
    --project neuiot \
    --devices s-echodot-1 s-googlenest-1 \
    --exp-name myexp \
    --exp-desc "Smart-speaker capture run" \
    --realization realiot \
    --duration 1w \
    --xdc myxdc
```

Inside the interactive control loop you can issue [`spiot_ctl` commands](#the-spiot_ctl-control-language):

```
spiot_ctl > exp devices
spiot_ctl > s-echodot-1 click_button
spiot_ctl > exit
```

To come back to an experiment `run` left behind, or to drive one from several
terminals, use the same prompt on its own:

```bash
mrg-iot cmd shell --project neuiot --experiment myexp --realization realiot --xdc myxdc
```

---

## Authentication

```bash
mrg-iot login                                  # prompts for username + password
mrg-iot login -u myuser                        # prompts for the password only
mrg-iot login -u myuser --password-stdin < f   # reads the password from stdin
mrg-iot logout                                 # clears the stored session
```

Credentials are written to `~/.mrg-iot/session.json` with mode `0600`. Every
authenticated command transparently re-validates the stored token and
re-prompts only if the portal *rejects* it — you rarely need to run `login`
more than once. If the portal simply can't be reached, the stored session is
kept rather than discarded, so a brief network problem doesn't cost you a
re-login.

You can also supply the username up front and let `mrg-iot` ask only for the
password:

```bash
mrg-iot login -u myuser          # prompts for the password only
```

### Unattended authentication (CI, cron, batch scripts)

Two options. Pipe the password in, keeping it out of your shell history and out
of the process list:

```bash
mrg-iot login -u myuser --password-stdin < ~/.secrets/portal
# or
printf '%s' "$PORTAL_PASSWORD" | mrg-iot login -u myuser --password-stdin
```

`login` and `logout` now exit non-zero on failure, so `|| exit 1` works.

> There is deliberately **no** `--password <pw>` flag. On Linux any user can
> read another process's arguments from `/proc/<pid>/cmdline`, and on every
> platform the command is recorded in shell history and echoed into CI logs.
> `--password-stdin` avoids all three.

Or supply credentials through the environment, which applies to *every*
subcommand rather than just `login` — no separate login step and no
`session.json` written:

```bash
# Either an existing bearer token...
export MRG_IOT_TOKEN=<token>
export MRG_IOT_USERNAME=<username>   # optional; used for display + SSH key lookup

# ...or a username and password, which are exchanged for a token normally.
export MRG_IOT_USERNAME=<username>
export MRG_IOT_PASSWORD=<password>

mrg-iot cmd run "exp devices"
```

`MRG_IOT_TOKEN` takes precedence and skips the login call entirely.
`MRG_IOT_USERNAME` + `MRG_IOT_PASSWORD` must both be set to take effect, and a
rejected password fails immediately rather than retrying (there is nothing to
re-prompt). If credentials are needed and none of these are available while
stdin is not a terminal, the command exits with an explanatory message rather
than a traceback.

> Prefer `MRG_IOT_TOKEN` in shared CI: a token can be revoked with
> `mrg-iot logout`, and it keeps the password out of the job configuration.

---

## How sessions work

Everything that talks to a running experiment -- sending commands, watching
cameras, moving files -- needs an SSH session to the XDC with tunnels forwarded
through it. **Each command opens its own session for as long as it runs, then
closes it.** There is no background daemon and nothing cached between
invocations.

Two consequences worth knowing up front:

- **Commands that give you something continuous have to keep running.**
  `stream open` holds its tunnel until you close the viewers. Closing the
  command closes the thing.
- **Sessions are cheap to run in parallel.** The local end of every tunnel is
  an OS-assigned ephemeral port, so you can have `stream open` in one terminal,
  `cmd shell` in another and `file download` in a third without colliding.

### Two required arguments, everything else derived

Which enclave a command targets comes from `--project / --experiment /
--realization / --xdc`. Only two of those are ever required, and the rest are
**derived, never asked for** — so every command in this README runs unattended,
in a script or in CI, exactly as written.

| Argument | Rule |
| --- | --- |
| `--project`, `-p` | Required only if you belong to more than one project. With one, it's used automatically; with several and no flag, the error lists them. |
| `--devices`, `-dev` | Required only on the commands that build a device model: `exp create`, `exp setup`, `realization create`, `run`. |
| `--experiment`, `-e` | Defaults to **your username** — `neuiotadm` gets the experiment `neuiotadm`. |
| `--xdc`, `-x` | Defaults to **`<experiment>xdc`** — so `exp setup myexp` provisions `myexpxdc`, and `-e myexp` alone is enough to reach it later. |
| `--realization`, `-r` | Defaults to **`realiot`**. |
| `--duration`, `-dur` / `--xdc-duration`, `-xdur` | Default to **`1w`** (minimum 4 days). |
| `--network`, `-net` | Defaults to **`mrg-iot-net`**. |
| `--description`, `-desc` | Defaults to `Experiment created with mrg-iot by <username>`. |

Every one of those is still overridable by its flag; the defaults are only what
you get when you omit it. So the shortest useful invocation is:

```bash
mrg-iot exp setup --devices s-echodot-1   # neuiotadm / realiot / neuiotadmxdc
mrg-iot cmd run "exp devices"             # …and this reaches the same enclave
```

`mrg-iot run` is the one exception: it is a guided flow that asks questions.
Pass `--non-interactive` to have it take the defaults above instead.

If a derived name points at something that doesn't exist, you get an error that
says which name it tried and which flag overrides it — the command never stops
to ask.

### One command for the whole lifecycle: `run`

`mrg-iot run` provisions an experiment, opens a session, drops you into the
command prompt, and tears everything down when you type `exit` (or send Ctrl-C
/ SIGTERM). It is the default if you type `mrg-iot` with no subcommand.

```bash
mrg-iot run                       # interactive, prompts for everything
mrg-iot run --simplified          # interactive but skips optional prompts
mrg-iot run --non-interactive ... # fully scripted (see Quick start)
```

Useful `run` flags:

| Flag | Purpose |
|------|---------|
| `--non-interactive`, `-n` | No prompts; all required args must be supplied. |
| `--simplified`, `-s` | Interactive, but uses sensible defaults for optional steps. |
| `--commands-file`, `-cf <path>` | Run a newline-delimited file of commands non-interactively. |
| `--download-files`, `-df` | Download the experiment archive on exit. |
| `--delete-xdc`, `-dx` | Delete the XDC during cleanup (default: keep). |
| `--delete-exp`, `-de` | Delete the experiment during cleanup (default: keep). |
| `--network`, `-net <name>` | Model network name (default: `mrg-iot-net`). |

> **Tip:** By default `run` *keeps* the XDC and experiment on exit, so you can
> come back to them with `mrg-iot cmd shell`. Pass `--delete-xdc`/`--delete-exp`
> for a full teardown.

### Step by step

The same lifecycle, decomposed -- useful when the experiment should outlive
your terminal, or when several people share one enclave.

```bash
# Create experiment + realization + materialization + XDC, and attach them
mrg-iot exp setup myexp --project neuiot --devices d1 d2 --xdc myxdc

# Use it, as often as you like, from any terminal
mrg-iot cmd shell     --project neuiot --experiment myexp --realization realiot --xdc myxdc
mrg-iot cmd run       "exp devices" "s-echodot-1 click_button"
mrg-iot stream open
mrg-iot file download --output ./traffic.zip

# Tear it all down
mrg-iot exp teardown myexp --project neuiot --realization realiot --xdc myxdc
```

`cmd shell` is the interactive prompt: one foreground session you keep open and
type into. `cmd run` runs a batch and exits.

### Commands that take time: `run`, `async-run`, and tasks

A device command either answers immediately or is **queued** — the experiment
takes the job, hands back an id, and does the work in the background:

```
Command 's-echodot-1 click_button' with id: 0, returned:
Command click_button is queued
```

`cmd run` finishes the job for you. When a reply says "queued" it sends
`query wait_result <device> <command> <id>` and prints the actual result, so one
command in means one result out however long the device takes:

```bash
mrg-iot cmd run "s-echodot-1 click_button"
# Click: Button Pusher Operation Successful
```

`cmd async-run` doesn't wait. It prints what came straight back and records the
job as a **task** with a handle of its own — `t1`, `t2`, … — which you pick up
in a later invocation:

```bash
mrg-iot cmd async-run "s-echodot-1 hold_button 10"
# Command hold_button is queued
# ✔ t1: s-echodot-1 hold_button (job 3)

mrg-iot cmd tasks          # everything on record, read straight off disk
mrg-iot cmd status t1      # query state: where it has got to
mrg-iot cmd wait t1        # query wait_result: block, then print the result
```

Handles are ours, not the experiment's. Job ids restart at 0 in every
experiment, so they are not unique on your machine and are never what you type;
a handle is unique everywhere and carries the project, experiment, realization
and XDC its job was started in. That is what makes `mrg-iot cmd wait t1` a
complete command — no flags, no prompts. A flag that contradicts the record is
refused rather than sent, since asking the wrong experiment about job 0 does not
fail, it answers about a different job.

`cmd tasks` opens nothing: no login, no portal call, no session. It deliberately
does not show each job's state, which would cost one round trip per row — that
is what `cmd status` is for. Both `--project` and `--experiment` narrow the list.

Tasks are forgotten when the experiment they belong to is torn down
(`exp undeploy`, `exp teardown`, `exp delete`), because a
redeployed experiment
starts counting job ids from 0 again and a surviving handle would ask about one
job and be answered about another.

`cmd run` normally records nothing — the job is done by the time it returns. The
exception is a `wait_result` that fails or times out: then the job really is
still running, so it is recorded and the handle printed rather than lost.

### From inside the XDC: `cmd ctl`

When you are already **inside an XDC**, none of the SSH/tunnel machinery is
needed -- the experiment's `ExperimentControl` server is reachable directly over
WireGuard. `mrg-iot cmd ctl` gives you the same prompt with no portal login and
no tunnel:

```bash
mrg-iot cmd ctl              # default host 192.168.254.1
mrg-iot cmd ctl 10.0.0.5     # override the ExperimentControl host
```

Each line you type is one `RunCommand` gRPC call; type `exit` (or Ctrl-C /
Ctrl-D) to quit. Prompt history is saved to `~/.spiot_history`.

### Removed spellings

The pre-split spellings have been **removed**. Each was a second name for a
subcommand that already existed, routed to the same handler, so there was
nothing it could do that its replacement could not:

| Removed | Use instead |
|---------|-------------|
| `mrg-iot send ...` | `mrg-iot cmd run ...` |
| `mrg-iot cmd send ...` | `mrg-iot cmd run ...` |
| `mrg-iot ctl` | `mrg-iot cmd ctl` |
| `mrg-iot show video` | `mrg-iot stream open` |
| `mrg-iot download traffic` | `mrg-iot file download` |

One caveat when updating a script: `cmd run` is not quite what `send` did.
`send` printed "Command click_button is queued" and exited, where `cmd run`
stays for the result. **`mrg-iot cmd async-run` is the closer match** to the
old behaviour.

`mrg-iot connect` and `mrg-iot disconnect` have been **removed**. They spawned a
background daemon that held one session open for every command to share; the
same job is now done by `mrg-iot cmd shell` (a foreground session you keep) or
by simply running each command on its own.

---

## Files

The XDC runs an SFTPGo file server that `mrg-iot` reaches through a tunnel.
Only its browser-facing web client is mounted (the REST API is not), so:

```bash
# What's there
mrg-iot file list

# Download one file, or the whole `output` directory when you name nothing
mrg-iot file download capture.pcap        # -> ~/Downloads/capture.pcap
mrg-iot file download                     # -> ~/Downloads/<enclave>.zip
mrg-iot file download capture.pcap --output /tmp/run.pcap

# Upload and delete
mrg-iot file upload ./script.py
mrg-iot file remove capture.pcap

# Or browse the file server yourself, in a browser
mrg-iot file client
```

Names are the file server's own, as `file list` prints them; all four commands
work in its root directory. `file download` with no name is the exception that
takes a directory: `output` can only come back through the web client's zip
endpoint, so it lands as `<enclave>.zip`.

Every one of these drives the web client programmatically -- parsing the
credentials out of the experiment's intro banner and logging in through its
form endpoints -- exactly as the `exit`-time download prompt in `run` does. No
`scp`/SSH access to the XDC is needed.

---

## The `spiot_ctl` control language

Whether you're in the interactive `run` loop or using `mrg-iot cmd run`, you speak
to devices through `spiot_ctl`. The built-in `help` command lists everything;
the most common verbs:

| Command | Description |
|---------|-------------|
| `help` | Show all available commands. |
| `exp devices` | List the devices in the experiment. |
| `exp cred <device>` | Get a device's credentials (e.g. `exp cred s-echodot-1`). |
| `exp sleep <seconds>` | Pause for N seconds. |
| `exp read <filepath>` | Read a command file on the XDC and run every line in it. |
| `<device> help` / `<device> commands` | Show the commands a device supports. |
| `<device> <command> help` | Detailed help for one device command. |
| `<device> [-s <id|result|state> <var>] <command> [args]` | Execute a device command (optionally storing its job id or result). |
| `query state <device> <command> <id>` | Check the state of a running command. |
| `query get_result <device> <command> <id>` | Fetch the result of a finished command. |
| `query wait_result <device> <command> <id>` | Block until a command finishes, then return its result. |
| `exit` | Tear down `spiot_ctl`, optionally download files, and clean up. |

The three `query` verbs are the ones you rarely need to type: `cmd run` sends
`query wait_result` for you when a command comes back queued, and `cmd wait` /
`cmd status` are those verbs addressed by task handle instead of by
`<device> <command> <id>`. See
[Commands that take time](#commands-that-take-time-run-async-run-and-tasks).

To discover what a specific device supports, ask the experiment once you're
connected: `<device> commands` lists its verbs and `<device> <command> help`
prints one verb's usage line. `mrg_iot/data/command_spec.json` in this repo
documents every command's declared argument types, ranges, and allowed values.

`mrg-iot` does **not** check these commands before sending them: the experiment
validates its own input, so a missing or out-of-range argument comes back from
the experiment rather than from the CLI. (mrg-iot still validates its *own*
arguments — experiment names, durations, and so on.) The one thing it does to a
line is rewrite the outdated `dev <device> <command>` form, which the experiment
now answers with "Invalid action type". The client-side checker is kept, commented
out at its two call sites — see `mrg_iot/core/command_check.py`.

---

## Command reference

`mrg-iot` exposes resource-CRUD subcommands so you can manage individual
testbed objects without running the full flow. Add `--help` to any subcommand
for its exact flags, and `--debug`/`-d` to any command for verbose logging.

### Experiments

The lifecycle verbs take the experiment name as a positional `NAME`
(defaulting to your username), so the whole flow can be driven one named
experiment at a time:

```bash
mrg-iot exp create   myexp --devices d1,d2   # experiment + model; nothing allocated yet
mrg-iot exp deploy   myexp                   # realize -> materialize -> xdc -> attach
mrg-iot exp status   myexp                   # created / realized / materialized / deployed
mrg-iot exp undeploy myexp                   # detach -> delete xdc -> delete mtz + rz
mrg-iot exp delete   myexp                   # remove the experiment record
```

The verbs pair up, and the two halves have a one-command spelling each:

```bash
mrg-iot exp setup    myexp --devices d1,d2   # create + deploy, in one command
mrg-iot exp teardown myexp                   # undeploy + delete, its inverse
```

`teardown` runs every step of `undeploy` before deleting the experiment, so it
leaves nothing allocated. An xdc or realization that is already gone is reported
as skipped, not as a failure — `exp undeploy && exp teardown` still exits 0.

`--devices`/`-dev` takes either spelling — `-dev d1 d2` or `-dev d1,d2` — and
duplicates are dropped. Because the flag is greedy, **`NAME` has to come before
it**: `exp create -dev d1 d2 myexp` reads as three devices and no name.

`deploy`, `setup`, `undeploy` and `teardown` also take `--realization/-r` and
`--xdc/-x`; `deploy` and `setup` add `--duration/-dur` and
`--xdc-duration/-xdur`; `create` and `setup` take `--description/-desc` and
`--network/-net`. All of them take `--project/-p`.
Anything omitted falls back to the usual defaults (`realiot`,
`<experiment>xdc`, `1w`, `mrg-iot-net`) — see
[Two required arguments](#two-required-arguments-everything-else-derived).

Inspection, and the older flag spellings, still work:

```bash
mrg-iot exp list [--project p] [--all]                      # yours; --all for everyone's
mrg-iot exp create --project p --name n --devices d1,d2 [--description d]
mrg-iot exp delete --project p --name n
```

### Realizations (alias: `rz`)

```bash
mrg-iot realization list   --project p --experiment e
mrg-iot realization status --project p --experiment e --realization r
mrg-iot realization create --project p --experiment e --devices d1 d2 \
        [--name r] [--network net] [--duration 1w]          # build model, compile, push, realize
mrg-iot realization delete --project p --experiment e --realization r
```

### Materializations (alias: `mtz`)

```bash
mrg-iot materialization list          --project p --experiment e
mrg-iot materialization status        --project p --experiment e --realization r
mrg-iot materialization create        --project p --experiment e --realization r
mrg-iot materialization delete        --project p --experiment e --realization r
```

### XDCs

```bash
mrg-iot xdc list [--project p]
mrg-iot xdc status --project p --name n
mrg-iot xdc create --project p [--name n] [--duration 1w] [--no-wait]
mrg-iot xdc attach --project p --name n --experiment e --realization r
mrg-iot xdc detach --project p --name n --experiment e --realization r
mrg-iot xdc delete --project p --name n
```

### Devices

```bash
mrg-iot devices list [--project p]      # devices free to use (omit --project for all)
```

Only *unallocated* devices are listed -- including hiding the ones your own
experiment holds. For the devices inside a running experiment, ask the
experiment: `mrg-iot cmd run "exp devices"`.

### Talking to a running experiment

Each of these opens its own session; all take
`--project / --experiment / --realization / --xdc`, and prompt for whatever you
omit.

```bash
mrg-iot cmd run "<cmd>" ["<cmd>" ...]   # run a batch, waiting for queued commands
mrg-iot cmd async-run "<cmd>" [...]     # don't wait; record each queued job as a task
mrg-iot cmd tasks [--project p]         # the jobs on record (local; opens nothing)
mrg-iot cmd wait TASK                   # block until that job finishes, print its result
mrg-iot cmd status TASK                 # just its state
mrg-iot cmd shell                       # interactive prompt
mrg-iot cmd ctl [host]                  # same prompt, from inside the XDC

mrg-iot stream open                     # VLC viewers; runs until they close
mrg-iot stream list                     # name the cameras
mrg-iot stream url                      # RTSP URLs; keeps forwarding until Ctrl+C

mrg-iot file list                       # what's in the experiment's directory
mrg-iot file download [name] [-o path]  # a file, or the whole output dir as a zip
mrg-iot file upload <path>              # put a local file in the experiment
mrg-iot file remove <name>              # delete one
mrg-iot file client                     # the web client, in a browser, until Ctrl+C
```

### Typical cleanup sequence

Usually just `mrg-iot exp teardown myexp`. The long form, when you need to drive
one resource at a time:

```bash
mrg-iot xdc detach --project neuiot --name myxdc --experiment myexp --realization realiot
mrg-iot xdc delete --project neuiot --name myxdc
mrg-iot materialization delete --project neuiot --experiment myexp --realization realiot
mrg-iot realization delete --project neuiot --experiment myexp --realization realiot
mrg-iot exp delete --project neuiot --name myexp
```

---

## Input validation rules

Inputs are validated up front so bad arguments fail fast with a clear
`[mrg-iot] Invalid <label>: …` message (exit code `2`) rather than deep inside a
portal call:

- **Names** (`exp-name`, `realization`, `xdc`, `network`): must match
  `^[a-z][a-z0-9]*$` — start with a lowercase letter, lowercase letters and
  digits only, max **32** chars.
- **Description**: letters, digits, spaces, commas, periods, hyphens; max
  **256** chars.
- **Duration**: minimum **4 days**. Accepts forms like `1w`, `4d`, `1w2d3h`,
  `1 week`. Anything under 4 days is rejected.

---

## Configuration & on-disk state

Everything user-scoped lives under `~/.mrg-iot/`:

| File | Purpose |
|------|---------|
| `session.json` | `{"username", "token"}` — your cached login (mode `0600`). |
| `debug.log` | Verbose CLI log, appended across runs and rotated at 5 MB (3 kept). Attach this to bug reports. |
| `tasks.json` | Background device jobs left running by `cmd async-run`, so `cmd wait`/`cmd status`/`cmd tasks` can name one in a later invocation (mode `0600`). Cleared per experiment when it is undeployed or deleted. |

That is nearly the whole of it. There is no cached *connection* state: which
enclave a command targets comes from its flags or from the interactive pickers,
every time. `tasks.json` is not an exception — it remembers what a job *is*
(device, command, job id, and the enclave it ran in), never a live connection. `connection.json`, `connection.sock` and `daemon.log` belonged to the
removed daemon and are safe to delete if an older version left them behind.

Environment variables that affect behaviour:

| Variable | Effect |
|----------|--------|
| `MRG_IOT_TOKEN` | Use this bearer token directly; skips login entirely. |
| `MRG_IOT_USERNAME` | Username for unattended login, and the account name shown/used for SSH key lookup. |
| `MRG_IOT_PASSWORD` | Password for unattended login (requires `MRG_IOT_USERNAME`). |

`--debug` raises the log level; logs are always written to
`~/.mrg-iot/debug.log` (appended across runs, rotated at 5 MB, 3 kept). Attach
that file and `mrg-iot --version` to any bug report.

### Endpoints & known projects

- **gRPC portal:** `grpc.sphere-testbed.net:443` (via `mergetbapi==1.3.41`)
- **SSH jump host:** `jump.sphere-testbed.net:2022`
- **Tunneled ports:** `8554` (RTSP), `9001` (SFTPGo file client — uploads/downloads), `17000` (`ExperimentControl`). Every local port is OS-assigned.
- **Known projects:** `neuiot`, `iotbeta`, `iotdev` (the hardcoded `PROJECTS` list in `mrg_iot/constants.py`; a project outside it is not listed)

---

## Architecture

Three tiers, with a dependency rule enforced by
`tests/test_import_layering.py`:

- **`core/`** — what more than one command needs: reaching an experiment (SSH,
  tunnels, session), naming which one to reach, and preparing a command line
  before sending it. Never imports a command group.
- **`commands/<group>/`** — one package per subcommand family. Each owns its own
  argparse subtree and handlers, and never imports another group.
- **`views/`** — the only place that writes to the terminal. Nothing outside it
  calls `print()` or touches `rich` directly.

```
mrg_iot/
├── cli.py                  # a registry: GROUPS + main() dispatcher
├── constants.py            # pure data: paths, endpoints, model template, help text
├── core/
│   ├── controller.py       # @require_login
│   ├── enclave.py          # interactive pickers + readiness checks
│   ├── ssh.py              # XDC SSH connect via the jump host
│   ├── tunnel.py           # Paramiko port forwarding (ephemeral local ports)
│   ├── command_check.py    # legacy-syntax rewrite; argument checking (disabled)
│   ├── auth.py             # credential precedence + login/logout
│   ├── api/                # portal gRPC client + retry wrapper
│   └── session/            # model, scope, channel, transport, checking, intro
├── commands/
│   ├── cmd/                # run | async-run | wait | status | tasks | shell | ctl
│   │                       #   (+ jobs.py: queued replies; the shared REPL loop)
│   ├── devices/            # list
│   ├── exp/                # exp | realization | materialization | xdc
│   ├── file/               # download | upload | list | remove (SFTPGo web client)
│   ├── session/            # login | logout
│   ├── stream/             # open | list | url  (+ the VLC viewer handling)
│   └── run.py              # the one-command lifecycle
├── models/                 # dataclasses + on-disk state
├── views/                  # all user-facing output
└── utils/                  # duration parsing, file helpers
```

**The `run` workflow** (`commands/run.py`, composing `commands/exp/provision.py`):

```
Login → Select Project → Select Devices → Create Experiment →
Build+Compile+Push Model → Realize → Materialize → Create/Attach XDC →
SSH + tunnels + ExperimentControl channel → command loop → Cleanup
```

Progress is recorded in an `ExpState` dataclass as each resource is created, so
cleanup tears down exactly what exists even when provisioning fails partway.

**Sessions.** `core.session.session_scope` is the single way to open one, for
every command. It is a context manager rather than an open/close pair
specifically so a Ctrl-C partway through building the tunnels cannot leak the
SSH client. Each local tunnel port is assigned by the OS, and URLs pointing at
forwarded services (RTSP, the file server) are built from the live session's
port map — there is no fixed port to hardcode.

---

## Batch device traffic testing

`run_device_tests.sh` drives `mrg-iot run` over a whole device inventory and
reports, per device, whether packets from that device actually appeared in the
experiment's capture — PASS / FAIL / SKIP plus a summary CSV.

```bash
EXP_NAME=bulbs ./run_device_tests.sh --repo bulbs_repo.yaml -n 20
```

Full operating guide, including the device repo format, batched mode, reruns of
just the failures, and the `--jobs` caveat:
[`docs/RUN_DEVICE_TESTS.md`](docs/RUN_DEVICE_TESTS.md).

---

## Troubleshooting

**SSL / TLS error talking to the portal (common on macOS):**

```bash
export SSL_CERT_FILE="$(python -m certifi)"
export REQUESTS_CA_BUNDLE="$SSL_CERT_FILE"
```

**`stream open` does nothing / errors:** Make sure the `vlc` binary is installed
and on your `PATH` (`vlc --version`). `pip`/`pipx` cannot install VLC. If you
would rather use another player, `mrg-iot stream url` prints the RTSP addresses
and keeps forwarding them.

**A file command can't find your file:** Names are the file server's, not your
shell's — no globbing and no paths. Run `mrg-iot file list` and use the name it
prints.

**VLC connects but shows nothing:** Check for an editor forwarding the same
port. VS Code's automatic port forwarding binds `127.0.0.1:<port>` and shadows
the tunnel for anything connecting to `localhost` — remove the entry from its
**Ports** panel, or set `"remote.autoForwardPorts": false`.

**A command hangs or times out:** Re-run with `--debug` and inspect
`~/.mrg-iot/debug.log` for the underlying gRPC/SSH error.

**Resources left behind after a crash:** Use the
[cleanup sequence](#typical-cleanup-sequence) to remove the XDC,
materialization, realization, and experiment individually.

---

## Development

Known gaps, open defects and the prioritised roadmap live in
[`docs/GAPS.md`](docs/GAPS.md) — read that before picking up work, and update it
when you close something.

```bash
git clone https://gitlab.com/sphere-neu/mrg-iot.git
cd mrg-iot
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
```

Tests, type-checking, linting, formatting:

```bash
pytest tests/ -v                              # asyncio_mode=auto; no extra flags
pytest tests/test_commands_cmd.py -v          # a single file
mypy mrg_iot
pylint mrg_iot
black mrg_iot tests && isort mrg_iot tests
```

> The `dev` extras (black, pylint, isort, etc.) require Python **3.10+**. On a
> 3.9 environment, install runtime deps only (`pip install -e .`).

Pinned dependency lockfiles live under `requirements/` (`requirements/prod.txt`
for runtime deps, `requirements/dev.txt` for the lint/format/type-check tools)
if you'd rather install from those than from `pyproject.toml`'s extras:

```bash
pip install -r requirements/prod.txt   # runtime only, Python 3.9+
pip install -r requirements/dev.txt    # runtime + dev tools, Python 3.10+
```

---

## Release process

CI (`.gitlab-ci.yml`) runs `pytest` on every push, builds an sdist + wheel on
success, and publishes to PyPI on `vX.Y.Z` tags via OIDC trusted publishing.

**The git tag must equal `v$(cat mrg_version)`** — the publish job aborts
otherwise. To cut a release:

```bash
# bump the version
echo "1.3.2" > mrg_version
git commit -am "increased version to 1.3.2"
git tag v1.3.2
git push && git push --tags
```

---

## License

[MIT](LICENSE) © Sphere / NEU IoT Testbed contributors
