# Runtime Sweep

You are executing the runtime-sweep skill. This verifies runtime registration and wiring integrity across the ONEX platform.

## Argument Parsing

```
/runtime_sweep [--dry-run] [--scope <omnidash-only|all-repos>]
```

```python
args = "$ARGUMENTS".split() if "$ARGUMENTS".strip() else []

dry_run = "--dry-run" in args
scope = "all-repos"  # default

if "--scope" in args:
    idx = args.index("--scope")
    if idx + 1 < len(args):
        scope = args[idx + 1]
        if scope not in ("omnidash-only", "all-repos"):
            print(f"Error: Invalid scope '{scope}'. Valid: omnidash-only, all-repos")
            exit(1)
```

## Announce

"I'm using the runtime-sweep skill to verify runtime registration and wiring integrity."

---

## Phase 1 — Node Description Audit

Scan all ONEX contract YAMLs across repos for node descriptions.

```bash
ONEX_REGISTRY_ROOT="${ONEX_REGISTRY_ROOT:?ONEX_REGISTRY_ROOT required}"  # local-path-ok shell variable for command examples

# Find all contract.yaml files under node directories
find "$ONEX_REGISTRY_ROOT"/*/src -name "contract.yaml" -path "*/nodes/*" 2>/dev/null  # local-path-ok command example using canonical repo path
```

For each contract, extract the `description` field. A description is flagged as placeholder if it:
- Matches the auto-generated pattern `compute+{hash}` or `Generated by generate_skill_node`
- Contains "TODO", "TBD", "PLACEHOLDER", or "FIXME" (case-insensitive)
- Is fewer than 10 characters
- Is identical to the node filename (e.g., `node_foo_bar` as the description)
- Is empty or null

Classify each node:
- `REAL`: meaningful description (>10 chars, not placeholder, not filename)
- `PLACEHOLDER`: matches any placeholder criteria above
- `MISSING`: no description field in contract

**Output table:**

| Contract Path | Node | Description Status | Description Preview |
|--------------|------|-------------------|-------------------|

---

## Phase 2 — Handler Wiring Audit

### Step 2a: List All Projection Files

```bash
OMNIDASH_PATH="${OMNIDASH_PATH:-${ONEX_REGISTRY_ROOT}/omnidash}"
ls "$OMNIDASH_PATH/server/projections/"*.ts 2>/dev/null | grep -v __tests__ | grep -v index
```

### Step 2b: Check Each Handler is Wired

For each projection file (e.g., `agent-routing-projection.ts`):

1. Extract the handler function it exports
2. Verify the handler is listed in `omnidash/topics.yaml` handler field:
```bash
grep -l "$HANDLER_NAME" "$OMNIDASH_PATH/topics.yaml"
```
3. Verify the handler is imported and called in `omnidash/server/read-model-consumer.ts`:
```bash
grep "$HANDLER_NAME" "$OMNIDASH_PATH/server/read-model-consumer.ts"
```

Classify:
- `WIRED`: handler in topics.yaml AND imported in consumer
- `UNWIRED`: handler exists but not in topics.yaml or consumer
- `ORPHAN_TOPIC`: topic in topics.yaml but no handler file

### Step 2c: Cross-Repo Handler Check (if --scope all-repos)

For each repo under `$ONEX_REGISTRY_ROOT/`, find contract YAMLs that declare `subscribe` or `publish` topics. Verify the handler referenced in the contract exists as an actual function.  # local-path-ok documentation reference to canonical repos root

```bash
find "$ONEX_REGISTRY_ROOT"/*/src -name "contract.yaml" -exec grep -l "subscribe\|publish" {} \;  # local-path-ok command example using canonical repo path
```

**Entry-point format rules — DO NOT flag these as malformed:**

- `module.path` (package form, no colon) — valid; omnimarket uses this exclusively
- `module.path:ClassName` (class form, colon-separated) — valid; other repos may use this
- Both forms are resolvable by `importlib.import_module`. Only flag a format error if import fails.
- `test_all_node_entry_points_are_package_form.py` enforces the omnimarket convention — never override it.

**Output table:**

| Handler File | Topic | Wiring Status |
|-------------|-------|---------------|

---

## Phase 3 — Topic Symmetry Audit

### Step 3a: Collect All Published Topics

Scan all contract YAMLs and topics.yaml files for topics listed under `publish` or `topics` keys:

```bash
# From contract YAMLs
grep -rh "publish_topics:" "$ONEX_REGISTRY_ROOT"/*/src/*/nodes/*/contract.yaml 2>/dev/null  # local-path-ok command example using canonical repo path

# From omnidash topics.yaml
grep "topic:" "$OMNIDASH_PATH/topics.yaml" 2>/dev/null
```

### Step 3b: Collect All Subscribed Topics

```bash
grep -rh "subscribe_topics:" "$ONEX_REGISTRY_ROOT"/*/src/*/nodes/*/contract.yaml 2>/dev/null  # local-path-ok command example using canonical repo path
grep "read_model_topics:" "$OMNIDASH_PATH/topics.yaml" 2>/dev/null
```

### Step 3c: Find Asymmetries

Compare published vs subscribed topic sets:
- `PRODUCER_ONLY`: topic has a publisher but no subscriber
- `CONSUMER_ONLY`: topic has a subscriber but no publisher
- `SYMMETRIC`: both sides exist

**Output table:**

| Topic | Producer | Consumer | Symmetry Status |
|-------|----------|----------|----------------|

---

## Phase 4 — Docker Log Analysis

For each running container, pull recent logs and scan for error patterns.

### Step 4a: List Running Containers

```bash
docker ps --format '{{.Names}}' 2>/dev/null | grep -E 'omnibase-infra-|omninode-'
```

### Step 4b: Pull and Scan Logs

For each container:

```bash
docker logs "$CONTAINER_NAME" --tail 100 2>&1
```

Scan for lines matching: `error`, `exception`, `failed`, `fatal`, `critical` (case-insensitive).

### Step 4c: Classify Containers

- `CLEAN`: 0 error lines in last 100 log lines
- `NOISY`: 1-5 error lines (may be transient)
- `ERROR_HEAVY`: >5 error lines — repeated errors indicate a real problem
- `CRASH_LOOP`: container state is `restarting` or restart count > 3

### Step 4d: Cross-Reference with Service Health

For each `ERROR_HEAVY` or `CRASH_LOOP` container:

```bash
docker inspect --format='{{.State.Health.Status}}' "$CONTAINER_NAME" 2>/dev/null
docker inspect --format='{{.RestartCount}}' "$CONTAINER_NAME" 2>/dev/null
```

Correlate:
- `ERROR_HEAVY` + `unhealthy` -> critical finding
- `ERROR_HEAVY` + `healthy` -> warn (errors not affecting health checks yet)
- `CRASH_LOOP` + any health -> critical finding

**Output table:**

| Container | Error Lines | Classification | Health Status | Restart Count |
|-----------|------------|----------------|---------------|---------------|

---

## Phase 5 — Live Container Crash-Loop Detection [OMN-9122]

Check `omninode-runtime*` containers for restart counts exceeding the threshold.

### Step 5a: Enumerate Runtime Containers

```bash
ONEX_INFRA_HOST="${ONEX_INFRA_HOST:-$(grep ONEX_INFRA_HOST ~/.omnibase/.env 2>/dev/null | cut -d= -f2)}"
ssh "jonah@${ONEX_INFRA_HOST}" "docker ps -a --format '{{.Names}}'" 2>/dev/null | grep '^omninode-runtime'
```

If SSH is unavailable, fall back to local Docker socket:

```bash
docker ps -a --format '{{.Names}}' 2>/dev/null | grep '^omninode-runtime'
```

### Step 5b: Check RestartCount for Each Container

For each `omninode-runtime*` container:

```bash
ONEX_INFRA_HOST="${ONEX_INFRA_HOST:-$(grep ONEX_INFRA_HOST ~/.omnibase/.env 2>/dev/null | cut -d= -f2)}"
ssh "jonah@${ONEX_INFRA_HOST}" "docker inspect --format='{{.RestartCount}}' $CONTAINER_NAME" 2>/dev/null
```

**Threshold:** `RestartCount > 5` → CRASH_LOOP_CRITICAL finding.

### Step 5c: Emit Friction and Fail if Threshold Exceeded

If any container has `RestartCount > 5`:

1. Collect all violating containers into a findings list.
2. Write a friction YAML file (see Phase 7 — Friction Output below).
3. **Exit non-zero** after completing all remaining phases and report output.

**Output table:**

| Container | RestartCount | Status |
|-----------|-------------|--------|
| omninode-runtime-foo | 0 | OK |
| omninode-runtime-bar | 7 | CRASH_LOOP_CRITICAL |

---

## Phase 6 — Orchestrator Consumer Group Health [OMN-9122]

Verify all orchestrator-owned Kafka consumer groups have at least one active member. A group in `Empty` state means handlers are not attached and events are silently dropped.

### Step 6a: Derive Orchestrator Consumer Groups from Contracts

Collect all `event_bus.subscribe_topics` entries from contract YAMLs for nodes whose names start with `omninode-runtime` or whose `owner` field matches the orchestrator service:

```bash
grep -rh "subscribe_topics\|consumer_group" \
  "$ONEX_REGISTRY_ROOT"/*/src/*/nodes/*/contract.yaml 2>/dev/null  # local-path-ok command example using canonical repo path
```

Extract the `consumer_group` value from each contract that declares `subscribe_topics`. If no explicit `consumer_group` is set, derive it as `onex-{node_name}`.

### Step 6b: Describe Each Consumer Group

For each consumer group derived in Step 6a:

```bash
ONEX_INFRA_HOST="${ONEX_INFRA_HOST:-$(grep ONEX_INFRA_HOST ~/.omnibase/.env 2>/dev/null | cut -d= -f2)}"
ssh "jonah@${ONEX_INFRA_HOST}" \
  "docker exec omnibase-infra-redpanda rpk group describe $GROUP_NAME --brokers localhost:9092" \
  2>/dev/null
```

Parse the output for `STATE` field:
- `STATE: Empty` → group has no active members → CONSUMER_GROUP_EMPTY finding
- `STATE: Stable` → healthy
- Command failure / group not found → CONSUMER_GROUP_MISSING finding

**Time-fence:** Only flag `Empty` state if the group was also `Empty` on the previous sweep run. To implement this, compare against the prior friction file if one exists under `$ONEX_STATE_DIR/friction/runtime_sweep_*.yaml`.

**Practical shorthand when previous sweep data is unavailable:** Flag any group currently in `Empty` state as CONSUMER_GROUP_EMPTY (conservative; safer than missing a stranded group).

### Step 6c: Emit Friction and Fail if Any Group is Empty

If any consumer group is in `Empty` or `Missing` state:

1. Collect all violating groups into a findings list.
2. Write a friction YAML file (see Phase 7 — Friction Output below).
3. **Exit non-zero** after completing all remaining phases and report output.

**Output table:**

| Consumer Group | State | Status |
|---------------|-------|--------|
| onex-node-orchestrator | Stable | OK |
| onex-node-effects | Empty | CONSUMER_GROUP_EMPTY |

---

## Phase 7 — Friction Output

If Phase 5 or Phase 6 found any violations, write a structured friction YAML to disk **before** exiting non-zero.

```bash
TIMESTAMP=$(date -u +%Y-%m-%dT%H-%M-%SZ)
FRICTION_DIR="${ONEX_STATE_DIR}/friction"
FRICTION_FILE="${FRICTION_DIR}/runtime_sweep_${TIMESTAMP}.yaml"
mkdir -p "$FRICTION_DIR"
```

Write the following structure (adapt fields to actual findings):

```yaml
skill: runtime_sweep
run_at: "<ISO8601 UTC timestamp>"
exit_code: 1
findings:
  crash_loop:
    - container: omninode-runtime-bar
      restart_count: 7
      threshold: 5
      status: CRASH_LOOP_CRITICAL
  consumer_group_empty:
    - group: onex-node-effects
      state: Empty
      status: CONSUMER_GROUP_EMPTY
```

Write atomically: write to `${FRICTION_FILE}.tmp`, then rename to `${FRICTION_FILE}`.

If no violations are found, do NOT write a friction file. Exit 0.

---

## Phase 8 — Report + Ticket Creation

### Summary Output

Print six summary tables:
1. **Node Descriptions** — per-contract description audit
2. **Handler Wiring** — per-handler wiring status
3. **Topic Symmetry** — per-topic producer/consumer symmetry
4. **Docker Log Health** — per-container log analysis
5. **Container Crash-Loop Health** — RestartCount per `omninode-runtime*` container
6. **Consumer Group Health** — state per orchestrator consumer group

### Ticket Creation

For each finding with status `PLACEHOLDER`, `UNWIRED`, `PRODUCER_ONLY`, `CONSUMER_ONLY`, `ERROR_HEAVY`, `CRASH_LOOP`, `CRASH_LOOP_CRITICAL`, or `CONSUMER_GROUP_EMPTY`:

**Only if `--dry-run` is NOT set:**

```
Title: fix(wiring): {finding_type} — {subject}
Project: Active Sprint
Labels: wiring, runtime-sweep
Description: include evidence (file paths, topic names, handler names, log excerpts)
```

Skip ticket creation for:
- `--dry-run` mode (print what WOULD be created instead)
- `REAL` / `WIRED` / `SYMMETRIC` / `CLEAN` / `NOISY` findings (healthy or transient)

---

## Dispatch Rules

- ALL work dispatched through `general-purpose`
- NEVER edit files directly from orchestrator context
- `--dry-run` produces zero side effects (no tickets)

## Integration Points

- **autopilot**: invoked as optional wiring verification step in close-out mode
- **data-flow-sweep**: complementary — data-flow checks live data; runtime-sweep checks static wiring
- **integration-sweep**: complementary — integration-sweep checks post-merge contracts; runtime-sweep checks registration
