Metadata-Version: 2.4
Name: nvidb
Version: 1.9.2
Summary: A package that provides an aggregated view of the NVIDIA GPU information on several hosts.
Author: L1ght
License-Expression: MIT
Project-URL: Homepage, https://github.com/FanBB2333/nvidb
Project-URL: Repository, https://github.com/FanBB2333/nvidb
Keywords: nvidia,gpu,monitoring,ssh,remote
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Hardware
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: paramiko
Requires-Dist: PyYAML
Requires-Dist: termcolor
Requires-Dist: nvidia-ml-py
Requires-Dist: blessed
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: setuptools; extra == "dev"
Dynamic: license-file

# nvidb

A package that provides an aggregated view of the NVIDIA GPU information on several hosts.

## 1. Installation

### 1.1 Install using `pip`

You can install `nvidb` using pip. First, clone the repository:

```bash
git clone https://github.com/FanBB2333/nvidb.git
cd nvidb
pip install .
```

Or install directly from PyPI:

```bash
pip install nvidb
# If the specified version is unavailable in your custom repository, use pypi.org as the source:
pip install nvidb -i https://pypi.org/simple
```

---

### 1.2 Configuration

#### Option A: Interactive Setup (Recommended)

Use the interactive command to add servers:

```bash
nvidb add
```

This will guide you through adding a new server with prompts for host, port, username, authentication method, etc.

#### Option B: Manual Configuration

To manually configure remote servers, create or edit the configuration file at `~/.nvidb/config.yml`:

```bash
mkdir -p ~/.nvidb/
cp config.example.yml ~/.nvidb/config.yml
# Edit the file with your server details
```

Configuration file [template](config.example.yml):

```yaml
servers:
  - hostname: "example1.com"       # Server hostname or IP address
    port: 22                       # SSH port number
    username: "user1"              # SSH username for authentication
    nickname: "Production GPU"     # Human-readable nickname for display
    auth: "auto"                   # Authentication method: auto | key | password
    identityfile: "~/.ssh/id_ed25519"  # Optional, used only when auth is auto/key
    proxyjump: "login.example.com" # Optional OpenSSH ProxyJump host or alias
    gpus: [0, 1]                    # Optional queue placement allowlist
```

**Configuration Options:**
- `hostname`: Server hostname or IP address (required)
- `port`: SSH port, default is 22 (required)
- `username`: SSH username (required)
- `nickname`: Human-readable server nickname (optional)
- `auth`: Authentication method - `auto`, `key`, or `password` (optional, default: `auto`)
- `identityfile`: SSH private key path (optional, only effective when `auth` is `auto` or `key`)
- `password`: SSH password (optional, will prompt if needed)
- `proxyjump`: OpenSSH jump host, alias, or comma-separated chain (optional; for
  example `login` or `login-a,alice@login-b:2222`)
- `gpus`: GPU indices available to new queue placements on this server
  (optional). Monitoring still shows every card, and running jobs outside the
  allowlist continue normally. An empty list disables GPU placement on the
  node while still permitting CPU-only queue jobs.

The server-level `gpus` list is a placement policy. The job option `--gpus N`
still means "request N cards"; it does not name individual indices.

> **Warning**: Storing passwords in plaintext in the configuration file is **NOT RECOMMENDED** for security reasons. Consider using SSH key-based authentication (`auth: key`) instead.

`proxyjump` uses the local OpenSSH client, so aliases and credentials configured
in `~/.ssh/config` are reused. Connect to the jump host once with `ssh login` to
accept its host key and verify key/agent authentication before running the
non-interactive job queue. `nvidb import` copies `ProxyJump` from OpenSSH config:

```sshconfig
Host login
    HostName login.example.com
    User jump-user

Host training-a100
    HostName 10.0.0.42
    User gpu-user
    ProxyJump login
```

```bash
nvidb import
nvidb --remote
```

The same file also holds a `view` section that nvidb maintains itself. Persistent
layout keys (`v`, `d`, `s`, `f`, `g`, `u`, `t`, and `p` in single-line mode)
write the new state back so the next run opens with the same view. Pane focus and
the temporary Detailed-mode process collapse are not persisted. See
[2.6 Interactive TUI Navigation](#26-interactive-tui-navigation) for what each
setting does.

#### Environment Variables

You can customize the working directory by setting `NVIDB_HOME`:

```bash
export NVIDB_HOME=/path/to/custom/nvidb
```

Default working directory is `~/.nvidb/`.

---

## 2. Usage

### 2.1 Basic Commands

```bash
nvidb                  # Monitor local GPU only (interactive TUI)
nvidb --remote         # Monitor local and remote servers
nvidb --once           # Print GPU stats once and exit
nvidb --once --remote  # Print all servers once and exit
nvidb --version        # Show version
```

> **Tip**: Set `remote: true` under the `basic` section of `~/.nvidb/config.yml` to make plain `nvidb` include remote servers by default (same for `nvidb log`). Pass `--no-remote` for a one-off local-only run.

GPU status is collected directly from NVML. Local collection uses `nvidia-ml-py`;
remote collection keeps a standard-library Python agent open over SSH and calls
`libnvidia-ml.so.1` directly, so no Python package needs to be installed on the
remote host. `nvidia-smi -q -x` is retained only as a compatibility fallback when
NVML cannot be initialized.

### 2.2 Server Management

```bash
nvidb add              # Interactively add a new server
nvidb import [path]    # Import servers from SSH config (default: ~/.ssh/config)
nvidb info             # Show configuration info and server list
```

### 2.3 GPU Logging

Continuously log GPU statistics to an SQLite database:

```bash
nvidb log                          # Log local GPU with default settings
nvidb log --remote                 # Log local and remote GPUs
nvidb log --interval 10            # Set logging interval to 10 seconds
nvidb log --db-path /path/to/db    # Specify custom database path
```

Press `Ctrl+C` to stop logging and save data.

### 2.4 Web Dashboard

Open a [Dash](https://github.com/plotly/dash)-based interactive web dashboard to view live GPU info and browse log sessions:

```bash
pip install dash
nvidb web                 # Web dashboard (Live + Logs)
nvidb web --db-path /path/to/db
nvidb web --port 8502
```

After the server starts (http://localhost:8501 by default):
- **Live**: per-server GPU tables plus rolling utilization / VRAM charts; toggle `include remote`, pick the refresh interval, or pause auto-refresh. (`basic.remote: true` or `nvidb --remote web` enables remote by default.)
- **Logs**: pick a session in the left table, then filter by node / metric / time range. Charts support zoom, pan and legend isolation; **click any chart point to inspect that snapshot**. The raw table supports filtering, sorting and CSV export.

`nvidb log web` is deprecated; use `nvidb web` instead.

### 2.5 Cleanup

Remove server configurations or delete log data:

```bash
nvidb clean              # Interactive cleanup menu
nvidb clean all          # Delete all data (requires double confirmation)
```

### 2.6 Interactive TUI Navigation

The monitor opens before connecting to remote nodes. Each node connects and
refreshes independently, so an unreachable or slow SSH server does not delay
healthy nodes. GPU rows appear first; CPU/memory and process details follow.
The live monitor does not query DCGM or show its advanced tables by default.
Use `nvidb --dcgm` (or `nvidb --remote --dcgm`) to explicitly enable DCGM
profiling; its metrics load after the base GPU rows. `--debug` does not enable
DCGM, and `d` only changes the GPU row layout.

By default, refresh progress is represented only by the
top **Updated** timestamp; per-node telemetry, loading/collecting phases, sample
ages and timing banners are hidden to keep the layout stable. Real connection
and collection errors remain visible.

Use `nvidb --debug` (or `nvidb --remote --debug`) to show collection phases,
stale-sample banners and per-node SSH/GPU/details timings (`v` switches to
per-node panels). Debug display is off on every normal invocation and is not
saved in the view settings. The `d` key still switches unified GPU row detail;
it does not toggle debugging.

The normal refresh interval is one second after each node finishes its own
sample; failed connections use increasing retry delays, including the first
failure. Password/passphrase prompts are still supported and temporarily pause
screen updates while other nodes continue collecting. `q` exits without waiting
for pending SSH calls (use Ctrl+C to exit a credential prompt).
ProxyJump helpers run in batch mode in the live monitor; jump hosts must already
have non-interactive authentication configured (for example, an unlocked key in
`ssh-agent`). Destination password/passphrase prompts remain available.

`--once`, logging and web consumers retain their complete-snapshot behavior.

When viewing GPU stats, use these keyboard shortcuts:

| Key               | Action                        |
| ----------------- | ----------------------------- |
| `v`               | Switch unified/per-node view  |
| `d`               | Toggle unified row detail     |
| `s`               | Cycle unified GPU sorting     |
| `f`               | Cycle unified GPU filters     |
| `g`               | Toggle unified per-node grouping |
| `u`               | Show/hide nodes without GPU support |
| `t`               | Toggle GPU and selected-process history |
| `Enter` / `Space` / `l` / `→` | Show and enter the selected GPU's process pane |
| `h` / `←`         | Return focus to the GPU/node pane |
| `Tab`             | Switch between visible GPU and process panes |
| `p`               | Show/hide the selected GPU's process pane |
| `j` / `↓`         | Move the active-pane selection down |
| `k` / `↑`         | Move the active-pane selection up |
| `PgUp` / `PgDn`   | Move the active-pane selection by a page |
| `[` / `]`         | Page through a long wrapped command |
| `/`               | Edit a live process filter |
| `o` / `F6`        | Cycle process sorting |
| `O`               | Reverse the current process sort |
| `+` / `-`         | Show more/fewer process rows |
| `i` / `T` / `K`   | Arm SIGINT / SIGTERM / SIGKILL for the selected process |
| `Esc`             | Cancel a signal or clear the process filter |
| `?`               | Open context-sensitive help |
| Mouse             | Select rows, sort headers, click actions, or scroll either pane |
| `Enter` (signal armed) | Confirm the pending process signal |
| `Enter` / `Space` (per-node view) | Toggle the selected server details |
| `a`               | Expand all servers            |
| `c`               | Collapse all servers          |
| `q`               | Quit                          |

The default per-node view keeps each server's summary and expandable detail
table. The unified view places GPUs from every node in one table. By default,
single-line rows are grouped into per-node blocks: a band names the node in
bold cyan followed by its hostname/IP, GPU count, free GPUs, average
utilization, and VRAM, with a dim rule filling the rest of the line. The rows
below it drop the redundant `Node` / `Hostname/IP` columns so more width goes
to the GPU metrics. Press `g` to turn grouping off (or sort by anything other
than node order) and the flat table with `Node` and `Hostname/IP` columns comes
back.
Detailed cards include node identity themselves and omit the redundant band.
Columns adapt to the terminal width, with core identity, utilization, model, and
VRAM fields kept ahead of secondary metrics.
In the unified view, press `d` to switch between the single-line table and
Detailed cards. Each card is divided into four labelled rows: `GPU` identity,
`LOAD`, `MEM/TEMP`, and `I/O`. Utilization and VRAM gain block bars on terminals
at least 100 columns wide. The palette stays deliberately quiet: grey for
structure and secondary values, cyan for the active focus and normal metrics,
and yellow/red for warnings. Green is reserved for a running/healthy state.
The node name comes before the GPU index so the machine is the first thing you
read.
PCIe throughput is scored against the link the card is actually running on. The
`PCIe` column names that link (`4.0x16`, `4.0x4`), and the `RX`/`TX` cells fill
up with their share of it — green under 40%, yellow past 40%, red past 70%, at
which point the interconnect is shaping transfers. Detailed cards spell the same
thing out as `Link 4.0x4 7.34GB/s of 4.0x16` plus a percentage per direction,
where the trailing `of ...` is the widest link the card itself supports. A card
wired through a narrower slot or riser reports the narrower link, and a card
sitting idle reports the reduced generation it drops to for power saving; both
are the real ceiling for the traffic measured alongside them. SXM boards are
scored on their host PCIe link, which is what these counters measure — NVLink
traffic between GPUs is not included.
The capacity line summarizes available and busy GPUs, average utilization, and
used/total/free VRAM. Press `s` to cycle between node order, available GPUs
first, and highest utilization first. A GPU is considered available when its
utilization is below 5% and its VRAM usage is below 10%.
Unified pages are sized from the current terminal height. The title shows the
current focus and visible GPU range, and `›` marks the active selection.
Press `f` to cycle through all, available, busy, and error-only views. Busy
means at least 50% GPU utilization. Error-only mode hides GPU rows and lists
nodes whose latest refresh failed. Because the filter is restored from the
config on the next run, a warning line above the table spells out how many GPUs
it is hiding.
Node failures also appear in a fixed top-of-body alert, ahead of GPU and
process details. Click that alert to open the error-only list; `j`/`k`,
PgUp/PgDn, and the wheel can reach every failed node on a short terminal.
Detailed mode places the selected GPU's process pane directly below the cards;
there is no separate drill-down screen. `Enter`, `Space`, `→`, or `l` shows the
pane when necessary and moves focus into its task list. Repeating one of these
keys keeps process focus instead of collapsing the pane. Press `p` to hide or
show it explicitly; `←` or `h` returns to GPU/node selection without hiding it.
The same navigation applies in single-line mode. The process table shows PID,
user, process VRAM, percentage of total GPU VRAM, CPU%, host MEM%, RSS, elapsed
time, state, and command. Narrow terminals discard secondary columns first. The
selected process gets an htop-style low-contrast grey background, and its block
below the table spells out both percentage of the whole GPU and percentage of
currently used GPU memory, threads, state, elapsed time, and the complete
wrapped command line.
Press `/` to filter the process list as you type. The query matches PID, user,
type, state, process name, and the full command. `Enter` keeps the current
filter and leaves edit mode; `Esc` clears it. Press `o` or `F6` to cycle through
VRAM, CPU, host memory, RSS, elapsed time, PID, and command sorting. Uppercase
`O` reverses the current order. The visible table headers are also clickable:
click a field once to sort by it and again to reverse it.
Use `+` and `-`, or the row controls in the action bar, to change how many
processes stay visible. The request is capped at 12 rows and reduced
automatically when the selected-process details need the space.
When a wrapped command would push the action bar off-screen, it automatically
uses height-aware pages (at most five command lines on terminals shorter than
28 rows). Use `[`/`]`, the clickable command buttons, or the wheel over the
command to see every line.

Use `Enter`, `→`, or `l` to enter process focus, `←` or `h` to return to
GPU/node focus, and `Tab` to switch between visible panes. Then use `j`/`k` or
the arrow keys to move the highlighted row. The active pane uses solid borders;
the inactive pane uses dashed borders. Only the active pane displays a
selected-row background, so process rows are not highlighted while GPU/node
selection has focus. Mouse reporting is on by default: click any GPU card or
process row to select it, use the wheel over either pane to scroll that pane,
and click the action buttons directly. Most terminals need `Shift` (or
`Option`) held down for their own drag-to-select while mouse reporting is
active; set `mouse: false` under `view` to disable TUI mouse handling.

The process action bar exposes `SIGINT`, `SIGTERM`, and `SIGKILL` as `i`, `T`,
and `K`. Every signal requires a second identical click/key press or `Enter`
within five seconds; `Esc` cancels it. Signals run on the node that owns the
selected GPU and report permission or command failures in the pane. If
GPU/node selection has focus, the first signal key only transfers focus to the
process pane; press it again to arm the action.

Press `?` for a help panel tailored to the current per-node, GPU, or process
focus. Click anywhere inside the panel, or press `?`, `Esc`, or `q`, to close
it. Closing help with `q` does not quit the monitor.

Press `t` (or click History) to show utilization, VRAM, and temperature for the
selected GPU plus CPU, GPU VRAM share, host memory, and RSS for the selected
process. Both histories retain the latest 60 successful refresh samples in
memory and do not add remote requests. On terminals shorter than 36 lines, the
history rows temporarily replace the selected process's command block so the
action buttons remain visible; toggle History off to restore the command.

The local machine is not one more node to manage: it reports through a
one-line strip at the very top of the page (in both views) with the system
description, CPU utilization, load average, and memory usage. It only keeps a
node section of its own while it has NVIDIA GPUs to show, or when it is the
only machine in the pool.

Remote machines without NVIDIA GPUs (a CPU-only host) are not expanded by
default in the per-node view and are collapsed into a single "hidden" line in
the unified node status. Press `u` to show them.
Layout toggles are written back to the `view` section of `~/.nvidb/config.yml`,
so the next `nvidb` run starts with the same layout. Process filters, sorting,
row counts, and help visibility are session-only.

### 2.7 GPU Monitor Decorator

Use the `@nvidb.monitor` decorator to track GPU usage during function execution:

```python
import nvidb

@nvidb.monitor
def train_model():
    # Your training code here
    pass

# With custom options
@nvidb.monitor(sample_interval=0.05, gpu_indices=[0, 1])
def multi_gpu_training(epochs: int = 100):
    pass

# Async function support
@nvidb.monitor
async def async_training():
    pass
```

After function execution, it outputs:
```
======================================================================
[nvidb.monitor] Function completed: train_model
  Signature: train_model()
  Location: /path/to/file.py:14
----------------------------------------------------------------------
  Duration: 125.3s
----------------------------------------------------------------------
  GPU 0: NVIDIA GeForce RTX 3090 Ti
    Memory:
      Peak:    8192.00 MiB / 24.00 GiB
      Delta:   +6144.00 MiB
    Utilization:
      Avg:     85.0%
    Temperature:
      Peak:    72C
    Power:
      Peak:    320.5W
======================================================================
```

**Decorator Options:**
- `sample_interval`: Sampling interval in seconds (default: 0.1)
- `gpu_indices`: List of GPU indices to monitor (default: all GPUs)
- `enabled`: Enable/disable monitoring (default: True)

---

## 3. Cluster Job Queue

`nvidb queue` adds a small slurm-like scheduler on top of the servers already in
`config.yml`. Jobs are submitted from any machine that can SSH to the nodes, wait
until a GPU has room for them, run detached on the node, and report back.

`nvidb q` is a short alias for `nvidb queue`; every queue subcommand accepts
either spelling.

The queue's only shared state is one SQLite file (`~/.nvidb/queue.db`). Several
independent clients — several Claude Code sessions, a script, an open TUI —
coordinate purely by reading and writing that file, so none of them has to stay
running for the queue to work.

### 3.1 How it schedules

There is no daemon. Any command that touches the queue performs a **tick**:
probe every node, settle finished jobs, then place whatever now fits. Ticks are
rate-limited and guarded by a lease, so a burst of concurrent clients results in
one round of SSH traffic rather than one per client. The open TUI ticks on a
timer, and `nvidb queue tick --watch 10` gives you a background ticker if you
want one.

GPUs are allocated by **VRAM budget**, not whole cards:

```
free = total − memory used by non-queue processes − reservations of queue jobs − headroom
```

Counting foreign processes is what lets the queue share machines with work you
started by hand: a card that someone else has filled simply reports no capacity,
and jobs go elsewhere until it frees up. A job is charged the larger of what it
reserved and what it actually uses, so understating `--vram` cannot oversubscribe
a card. CPU-only jobs (`--gpus 0`) reserve nothing, so they are bounded by a
count instead: `max_cpu_jobs_per_node`, four by default.

A job cancelled or timed out while its node was unreachable keeps its record
final but its process alive. The pid is remembered separately from the job — it
has to outlive both `job requeue` and `job purge` — and the first probe that
reaches that node again kills it, provided the pid is still running that job's
`run.sh`. `nvidb queue status --json` reports how many such cleanups are
outstanding as `pending_reaps`.

On nodes where the driver accounts for none of the memory in use — WSL, where
the GPU is driven from Windows, and which either names no processes at all or
names them without memory figures — the split between foreign work and the
queue's own jobs cannot be measured. Those GPUs are marked `blind` and their own
jobs are credited up to their reservation, which keeps the queue from charging a
job twice.

### 3.2 What is on the GPUs

The queue's own bookkeeping is only half the picture: these are workstations,
and most of what runs on them was started by hand. `nvidb queue nodes` reports
the whole card.

```bash
nvidb queue nodes               # capacity, plus the unmanaged work behind it
nvidb queue nodes --procs       # every process on every GPU, with its owner
nvidb queue nodes --json        # the same as structured data
```

```
  workstation  [up]  10.0.0.2  seen 00:00:00 ago
    GPU   MODEL                       UTIL  MEM              UNMANAGED      QUEUE  RESERVED  JOBS  FREE
    GPU0  NVIDIA GeForce RTX 3090 Ti  10%   23.2G/24.0G 97%  15.2G (blind)  0M     8.0G      1     316M
      unmanaged  ~15.2G of 23.2G in use; this driver reports no per-process memory
        pid 1608997  /python3.11  alice  job 12 sweep
```

`UTIL` and `MEM` describe the card itself, whoever is using it. `UNMANAGED` is
what the queue did not start — `(2p)` counts the processes behind it, `(blind)`
means the figure is inferred rather than measured. `QUEUE` is what the queue's
own jobs hold right now, against the `RESERVED` they were promised.

In JSON each GPU carries `mem_used_mb`, `mem_total_mb`, `mem_used_percent`,
`util_percent`, `external_mem_mb`, `queue_mem_mb`, `reserved_mb`, `free_mb`, and
a `processes` list naming every process with `managed` and `job_id`.

### 3.3 Submitting and watching jobs

```bash
# Put the command last, after `--`, or pass it as one quoted string
nvidb job submit --name train --vram 20G -- python train.py --epochs 10
nvidb job submit --node gpu-node --vram 8G --workdir ~/proj -- python eval.py
nvidb job submit --gpus 0 -- python prepare_data.py        # CPU-only
nvidb job submit --after 12 --vram 4G -- python report.py  # runs after job 12
nvidb job submit --script run.sh --vram 12G                # a local file's contents

nvidb job ls                    # everything, newest state
nvidb job ls --active           # only pending and running
nvidb job show 12 --logs 40     # detail plus the tail of stdout
nvidb job logs 12 -f            # follow the output
nvidb job wait 12 13            # block until both finish (0 ok, 1 failed, 2 timed out)
nvidb job edit 12 --vram 39G    # change a pending job's reservation in place
nvidb job cancel 12             # kill the remote process group
nvidb job requeue 12            # run a finished job again
nvidb job priority 12 5         # set the priority (bare number sets, +1/-2 adjust)
nvidb job priority 12 --up 2    # move a pending job two slots earlier instead
nvidb job purge                 # forget finished job records

nvidb queue status              # nodes, capacity and jobs in one view
nvidb queue status --procs      # the same, with every GPU process listed
nvidb queue events --since 42   # replay what happened while you were away
nvidb queue drain gpu-node        # stop scheduling onto a node (`resume` undoes it);
                                  # jobs already on it keep running and still report back
nvidb queue ignore offline-node   # stop probing/scheduling and hide a disconnected node
nvidb queue unignore offline-node # restore it; `nodes --include-ignored` lists hidden nodes
nvidb queue tick                # force one scheduler pass
```

Useful `submit` options: `--priority N` (higher goes first), `--timeout SECONDS`
(kill an overrunning job), `--retries N` (restart if the process vanishes),
`--env KEY=VALUE`, `--tag`, `--note`, and `--wait` to block until the job
finishes.

`job edit --vram` changes the reservation of a `pending` job without replacing
its record, so its lane/queue position, note and submitter are preserved. The
command runs a scheduler pass afterward unless `--no-tick` is given. A running
job's reservation cannot be changed because its placement is already active.

Read commands refresh the queue themselves before printing, so `nvidb job show`
never reports stale state. Pass `--no-tick` for a pure database read, or
`--tick` to force a refresh.

Every job runs with `CUDA_VISIBLE_DEVICES` set to its allocation, plus
`NVIDB_JOB_ID`, `NVIDB_JOB_NAME`, `NVIDB_NODE`, `NVIDB_JOB_DIR`, and
`NVIDB_STATUS_FILE`.

### 3.4 Notes and live progress

A job carries two independent pieces of free text, kept apart because they have
different authors and neither should be able to overwrite the other:

**`note`** — what you or a client says *about* the job. Set it at submit time and
edit it whenever, including long after the job has finished:

```bash
nvidb job submit --note "baseline A, lr=1e-4" -- python train.py
nvidb job note 12                              # read it
nvidb job note 12 --append "loss plateaued at epoch 30"
nvidb job note 12 "superseded by job 19"       # replace
nvidb job note 12 --clear
```

**`progress`** — what the job says about *itself*. The job writes one line to
`$NVIDB_STATUS_FILE` and the scheduler collects it on every probe, so you can
watch a long run without tailing its log:

```python
# inside a training script
import os
with open(os.environ["NVIDB_STATUS_FILE"], "w") as handle:
    handle.write(f"epoch {epoch}/{total} loss {loss:.3f}")
```

```bash
echo "epoch $i/$n loss $loss" > "$NVIDB_STATUS_FILE"   # or from shell
```

Only the last line of the file is read, so overwriting is the normal pattern.
Both fields appear in `job ls`, `queue status`, `job show`, the TUI and every
`--json` payload. The last thing a job reported is kept on its finished record,
which is often the quickest explanation of how far a lost or failed job got. A
retry clears the stale status line but keeps your note.

### 3.5 Failures, alerts and the optional daemon

When something goes wrong on a node, the queue records an **alert** locally,
classified by what actually happened:

| Kind                | Raised when                                          | Severity |
| ------------------- | ---------------------------------------------------- | -------- |
| `job_failed`        | the command exited non-zero                          | error    |
| `job_lost`          | the process vanished with no exit status             | error    |
| `job_timeout`       | the job passed `--timeout` and was killed            | error    |
| `dependency_failed` | a job can never run because a dependency failed      | error    |
| `job_unschedulable` | no GPU in the cluster is big enough to ever hold it  | error    |
| `job_retried`       | the process vanished but a retry remains             | warning  |
| `launch_failed`     | the job could not be started (disk full, permissions)| warning  |
| `node_down`         | a node stopped answering                             | error    |

For a failed job the queue pulls the tail of its stderr (falling back to stdout)
onto the alert, so the reason is readable locally without another round trip:

```bash
nvidb queue alerts            # what needs attention
nvidb queue alerts --detail   # ... with the captured output
nvidb queue ack 3             # acknowledge one
nvidb queue ack --all         # acknowledge everything
```

`nvidb queue alerts` exits non-zero while anything is unacknowledged, so a shell
or an agent can branch on it. Alerts stay until acknowledged; they are never
re-raised for the same failure, and `nvidb queue status` and the TUI both lead
with them.

Recording an alert and delivering it are separate. Recording happens on whatever
scheduler pass notices the failure, so nothing is lost when nobody is watching.
Delivery is the daemon's job:

```bash
nvidb queue daemon                      # tick every 15s and push failures
nvidb queue daemon --interval 5
nvidb queue daemon --once --json        # a single pass, for cron
```

The daemon is **optional** — the queue works exactly as before without it, since
every command runs a scheduler pass. Run it when you want failures pushed to you
rather than waiting to be asked, and for prompt timeout enforcement: without it,
a job that overruns is only killed the next time some command happens to tick.

Each alert is delivered once, whether or not the daemon restarts. Channels are
configured under `queue.notify`: a desktop notification, a JSON-lines file at
`$NVIDB_HOME/alerts.log`, and a `command` hook that receives the alert as JSON
on stdin — which is how you route failures to anything else.

### 3.6 Driving the queue from other programs

Every command accepts `--json` and prints one JSON document on stdout, which is
the intended way for tools — Claude Code sessions in particular — to use the
queue:

```bash
nvidb queue status --json     # nodes, per-GPU budgets, job table, counts
nvidb job submit --json --vram 20G -- python train.py
nvidb job wait 12 --json --logs 40
nvidb queue events --json --since 42
```

Three things make the database usable as a coordination channel between
processes that never talk directly:

- **Dependencies.** `--after 12,13` records the ordering in the queue, so a
  client can lay out a pipeline and exit; the jobs still run in order.
- **Results.** A job that writes `$NVIDB_JOB_DIR/result.json` has that payload
  collected on completion and served by `nvidb job result <id>`. Clients can
  also write one directly with `nvidb job result <id> --set '{"note":"..."}'`,
  which passes structured data between them without a shared filesystem.
- **Events.** `nvidb queue events --since <id>` replays every state change, so a
  client that was not running can catch up on exactly what it missed.

#### Agent skill

[`skills/nvidb-queue/`](skills/nvidb-queue/SKILL.md) is an Agent Skill that
teaches Claude Code, Codex and other skill-aware tools to route GPU work through
the queue instead of starting it over raw SSH. Install it by symlinking, so both
agents track the repository:

```bash
ln -s "$PWD/skills/nvidb-queue" ~/.claude/skills/nvidb-queue
ln -s "$PWD/skills/nvidb-queue" ~/.codex/skills/nvidb-queue
```

### 3.7 The queue TUI

```bash
nvidb queue                 # or: nvidb queue tui
```

The screen opens with a task-flow pane above the job table and detail or log
pane. It follows the active server configuration order (the same list as the
monitor TUI unless `queue.yml` deliberately overrides `servers`); database rows
for removed servers do not linger in the normal view. Every GPU gets a row that
reads from left to right: live utilisation and free VRAM, the running job, the
number of jobs in that card's lane, and as many ordered job cards as fit:

```text
G0  30% · 23.5G free  [ RUN #10 train 01:00:00 ] ──▶ { lane Q3 } ──▶ [1:#11 P+5 eval ←✓#10] ──▶ [2:#12 P0 report ←◇#11]  +1
ANY GPU ──▶ { Q2 · priority } ──▶ [1:#20 P+8 urgent]
DEPS #10 ─✓▶ #11 · #11 ─◇▶ #12
```

`lane Q3` is the queue attached to that exact card; the numbered positions are
its real execution order. `P+5` is the job priority (it controls dispatch in the
shared `ANY GPU` pool and is informational once a job has a fixed lane).
`✓` dependencies require successful completion, while `◇` dependencies accept
any terminal result. Held jobs and paused or blocked lanes are called out on
their GPU row. Pending GPU jobs without a lane are deliberately kept in the
`ANY GPU` pool because the scheduler has not assigned them to a card yet.

The GPU activity badge distinguishes queue-managed `RUN`, `EXTERNAL` work,
unattributed `BUSY` activity, confirmed `IDLE`, and `UNKNOWN` readings (including
unreachable nodes). Clicking `lane QN`, the shared pool, or a `+N` overflow count
opens the complete pending list for that pool. Job detail includes the lane
position and each dependency's state, even when the upstream job is older than
the recent-job list. Commands and working directories wrap across detail pages.

Press `v` or click the view control to switch to the server-capacity pane with
memory bars and process details. All SSH work happens on a worker thread, so an
unreachable node slows the numbers down but never freezes the interface.

| Key                | Action                                        |
| ------------------ | --------------------------------------------- |
| `j` / `k` / ↑ / ↓ | Move the node or job selection                |
| `←` / `→`         | Select a GPU on the focused node              |
| `PgUp` / `PgDn`    | Scroll resource rows, or page job selection   |
| `Tab`              | Switch focus between the node and job panes   |
| `Enter`            | Scope to selected node/GPU, or toggle detail  |
| `[` / `]`          | Page through wrapped detail or log text       |
| `L`                | Toggle a live tail of the selected job's log  |
| `c`                | Cancel the selected job (press twice)         |
| `r`                | Re-queue the selected finished job            |
| `+` / `-`          | Raise or lower the selected job's priority    |
| `K` / `J`          | Move a pending job up/down the dispatch order |
| `s` / `S`          | Cycle the sort column / flip its direction    |
| `t`                | Force a scheduler tick now                    |
| `a`                | Toggle automatic ticking                      |
| `f`                | Cycle the job filter                          |
| `x` / `Esc`        | Clear the current server/GPU/task-pool scope  |
| `v`                | Switch task flow / server capacity view       |
| `p`                | Open capacity view; cycle GPU process detail  |
| `d`                | Drain or resume the selected node             |
| `A`                | Acknowledge every open alert                  |
| `?`                | Help                                          |
| `q`                | Quit                                          |
| Mouse click        | Scope server/GPU jobs, select, sort or act    |
| Mouse wheel        | Scroll resources/help, jobs, or detail/log    |

The job table opens in *runtime* order, longest-running first; jobs without a
runtime follow the measured jobs. Queue order remains available through `s`,
and using `K`/`J` automatically restores it before moving a pending job.
For lane jobs, queue sorting follows lane position and `K`/`J` moves the job
within its lane without changing priorities. Lanes are grouped independently;
there is no single dispatch order across GPUs. Free-pool reordering adjusts
priorities, and the `PRI` column always shows the real values. Clicking a column header sorts
by that column (a second click flips it). In the server-capacity view, each GPU
line draws one memory bar whose segments distinguish foreign memory (amber)
from this queue's reservations (teal) and free space (dim). Colours are
deliberately muted: healthy values render grey or plain, and saturation is
reserved for states that need attention.

Mouse reporting is enabled by default. Clicking a server limits the lower table
to jobs running on that server; clicking any GPU cell drills down to jobs whose
allocation includes that card. Clicking a running or queued task card in the
flow opens that job directly; clicking it again hides its detail. `[all jobs]`,
`x`, or `Esc` returns to the global job view. Click a job-table row to select it;
clicking the selected row again shows or hides its detail pane. The wheel acts
on the pane under the pointer. Status counts select their matching global job
filter, and clicking a job alert opens that job's log. The bottom action bar
exposes the actions that apply to the current selection, including the second
confirmation required for cancellation. Most terminals reserve normal text
selection for `Shift`-drag or `Option`-drag while mouse reporting is active. Set
`mouse: false` under `view` in `~/.nvidb/config.yml` to disable it in both nvidb
TUIs.

The resource pane scrolls independently, so GPU rows and the shared/dependency
pools remain reachable even when a single node is taller than the pane. In
help, use `j`/`k`, PgUp/PgDn, or the wheel to read every command; `g`/`G` jumps
to the first/last help page.

### 3.8 What runs on the nodes

Nothing is installed. A generated `run.sh` is delivered over SSH and started
with `setsid`, so the job outlives the client that launched it. Each job keeps a
directory on its node (`~/.nvidb/jobs/<id>/` by default) holding the script,
`stdout.log`, `stderr.log`, its pid, and the exit status with the time it
finished. Every later interaction — checking liveness, reading output, killing a
job — is a single shell round trip.

Tuning lives under `queue:` in `config.yml`; see
[config.example.yml](config.example.yml) for the full set of keys.

### 3.9 Keeping the queue on a machine that stays on

Everything above assumes some client eventually runs a scheduler pass. That
stops being enough when the point is to submit work and walk away: a job queued
just before a laptop is closed stays `pending` until something looks again.
Jobs already **running** are unaffected — they were detached from the SSH
session that started them — so the only thing missing is somebody to dispatch.

Move the queue onto a machine that stays on, and the laptop becomes a client of
it. The queue's own configuration lives in its own file, so one `~/.nvidb` can
hold both a monitor watching many machines and a queue scheduling onto a few:

```yaml
# ~/.nvidb/queue.yml on the queue host — see queue.example.yml
servers:
  - hostname: "gpu-node.example.com"
    port: 2222
    username: "user"
    nickname: "gpu-node"
    gpus: [0, 1]  # optional: only new queue jobs are restricted
queue:
  include_local: true
  local_node_name: "queue-host"
```

The **keeper** keeps that machine scheduling. Its default shell supervisor is
written into `~/.nvidb`, restarts `nvidb queue daemon` whenever it stops, and
detaches from the session that installed it — no root, cron entry, or service
manager:

```bash
nvidb queue keeper install --start   # write ~/.nvidb/queue-keeper.sh and run it
nvidb queue keeper status            # non-zero while nothing is keeping the queue moving
nvidb queue keeper logs -n 50
nvidb queue keeper stop
```

On a Linux host with a working user systemd session, install a user service
instead. `--start` enables and starts `nvidb-queue.service`; systemd then
restarts the daemon after failures and starts it with the user's next session:

```bash
nvidb queue keeper install --systemd --start
systemctl --user status nvidb-queue.service
```

Starting the user service during boot, before that user logs in, additionally
requires lingering where the host permits it:

```bash
loginctl enable-linger "$USER"
```

The shell supervisor does not survive a reboot. Any later client command starts
it again. A client configured with `remote:` opens no database of its own and
forwards the whole command line to the queue host, running the keeper's `ensure`
in the same round trip. A missing or failed keeper is reported on stderr rather
than silently leaving the queue idle:

```yaml
# ~/.nvidb/queue.yml on the laptop
remote:
  host: "queue-host.example.com"
  nvidb: "/home/user/.local/bin/nvidb"   # absolute: a non-interactive shell has a thin PATH
```

`nvidb job submit`, `job wait`, `logs --follow`, `queue status` and the TUI then
behave exactly as they do locally — it is the same CLI, running over there — and
the exit codes come back unchanged. `--local` runs one command against this
machine's own queue instead, and `nvidb queue status` reports `keeper up` or
`keeper DOWN` wherever a keeper is installed.

Back up the queue with SQLite's online backup API, which includes committed WAL
state without pausing the daemon:

```bash
nvidb queue backup                         # ~/.nvidb/backups/queue-<time>.db
nvidb queue backup /secure/queue-copy.db  # explicit destination, never overwritten
```

The daemon can create and rotate backups automatically:

```yaml
queue:
  backup:
    enabled: true
    interval_hours: 24
    keep: 7
    directory: null        # defaults to $NVIDB_HOME/backups
```

Each snapshot is checked with `PRAGMA quick_check` before its complete temporary
database is atomically published without replacing an existing path.
`nvidb queue status --json` includes `last_backup_at`.

---

## 4. System Requirements

- NVIDIA driver with NVML (`libnvidia-ml.so.1`)
- Python 3.9+
- Python 3.9+ and SSH access on remote servers
- Local OpenSSH client when `proxyjump` is configured
- `nvidia-smi` is optional and used only as an NVML failure fallback

## 5. Tips

- The live header shows `Source: nvml` during normal collection and
  `Source: nvidia-smi` if the compatibility fallback was needed
- Database files are stored in `~/.nvidb/gpu_log.db` by default
- Configuration and logs are stored in `~/.nvidb/` directory

## 6. Show me the screenshots

- Monitor local info with `nvidb`:

![nvidb local](resources/nvidb_local.png)

- Monitor remote info with `nvidb --remote`:

![nvidb remote](resources/nvidb_remote.png)

- Monitor on web panel with `nvidb web`:

Local info:

![nvidb web local](resources/nvidb_web_local.png)

Remote info:

![nvidb web remote](resources/nvidb_web_remote.png)

---

## 7. Acknowledgements

- Thanks to NVIDIA for providing NVML and
  [nvidia-ml-py](https://pypi.org/project/nvidia-ml-py), used for direct GPU
  telemetry collection.
- Thanks to [nvitop](https://github.com/XuehaiPan/nvitop) for demonstrating
  efficient direct NVML polling and metric caching patterns.
- Thanks to NVIDIA for providing `nvidia-smi`, retained as a compatibility
  fallback.
- Thanks to [Paramiko](https://github.com/paramiko/paramiko) for powering SSH connections for remote monitoring.
- Thanks to [PyYAML](https://github.com/yaml/pyyaml) for YAML-based configuration loading and saving.
- Thanks to [pandas](https://github.com/pandas-dev/pandas) for parsing and processing GPU stats and log data.
- Thanks to [blessed](https://github.com/jquast/blessed) for building the interactive terminal UI.
- Thanks to [termcolor](https://github.com/termcolor/termcolor) for colored terminal output.
- Thanks to [Dash](https://github.com/plotly/dash) and [Plotly](https://github.com/plotly/plotly.py) for powering the web dashboard.
