# spike — Nemotron 3.5 Lightning SERVES on the Jetson AGX Thor on vLLM v0.29.0, up to the native 1M window
#
# Measure-only (operator directive, 2026-09-13). Nothing here changes a shape, a role, a
# profile, or the gateway; no box's deployed lobes configuration was modified. The worker lane
# (model-gear-vllm-worker) was STOPPED for the session with operator approval and is NOT
# restarted by this transcript — restarting it awaits explicit operator approval.
# Frame: .devague lightning-on-thor (scope entries s1-s19). Raw result files:
# thor:~/lightning-spike/{results,results-dspark,results-bf16}/ and boot logs beside them.
#
# Reverses docs/evidence/2026-08-20-spike-lightning-thor-no-go.txt for THIS engine:
# the Mamba2 SSD Triton warmup that wedged for 25+ min on nightly 8bd082 (0.26.1rc1) and on
# v0.27.1 completes in ~91 s on vllm/vllm-openai:v0.29.0-aarch64 (4 boots out of 4).
#
# ============================== SUMMARY ==============================
#
# Engine    vllm/vllm-openai@sha256:18372a7224938643461b846fb64c5c9d3d6e9727e82caf2dc3043e620c9d4d7a
#           (v0.29.0-aarch64, torch 2.13.0+cu130, sm_110 in arch list; verified live)
# Model     nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (snapshot bee7596271d1), util 0.7,
#           recipe flags (flashinfer Mamba backend, float16 SSM cache, stochastic rounding,
#           philox 5, cache mode align, prefix caching, 16384 batched tokens), bridge network,
#           port bound to 127.0.0.1 only (never --network host).
#
# Runs      A  plain decode, fp8 KV, 128000        (boot + probes)
#           B  plain decode, fp8 KV, 1,048,576      KV pool 20,431,534 tokens (19.49x at 1M = KV-pool capacity ceiling)
#           C  DSpark x5,    fp8 KV, 1,048,576      KV pool 17,911,473 tokens (17.08x at 1M = KV-pool capacity ceiling)
#           D  plain decode, bf16 KV, 1,048,576     KV pool 10,392,269 tokens (9.91x at 1M = KV-pool capacity ceiling)
#
# CONCURRENCY RATIOS: every "Nx at 1M" / "Maximum concurrency ... Nx" figure below is vLLM's
#   KV-pool CAPACITY CEILING (KV cache tokens divided by max_model_len) — how many full-length
#   requests the cache could hold. It is NOT measured concurrency and must NOT be multiplied by
#   single-stream tok/s to project aggregate throughput. Measured aggregate throughput is the
#   N=1-32 concurrency sweep (e.g. N=32: 541 tok/s total, 17 tok/s per request, fp8 plain).
#
# Correctness: every probe (known-answer, multi-step, tool call), every needle (~128K, ~250K,
#   1,040,073 tokens = 99.2% of the window), 180/180 agentic tool calls and 45/45 cross-context
#   recalls PASSED in all of B, C and D. No accuracy difference attributable to the
#   uncalibrated fp8 KV scales was found by this probe set.
#
# Which configuration wins, by workload (details and caveats in the sections below):
#   single-stream prose        B fp8 plain  (~79 tok/s; D ~74; C ~83 but noisy 77-87)
#   single-stream code         C DSpark     (141 tok/s = 1.79x) — out of associate's contract
#   decode at 128K / 250K      B fp8 plain  (70.8 / 66.7 tok/s; C 67.5 / 66.1; D 61.2 / 52.7)
#   decode at 1M depth         C DSpark     (45.0 tok/s; B 33.4; D 26.8)
#   cold prefill, every depth  D bf16       (1M: 1,449 s under throttle alerts; B 1,688; C 1,661)
#   concurrency 1..32          B ~= D > C   (N=32 aggregate 541 / 528 / 517 tok/s; no knee)
#   agentic, 1 session         D bf16       (wall 47.8 s; B 54.7; C 65.1)
#   agentic, 4 sessions        D bf16       (wall 138.6 s; B 153.6; C 184.7; C had a 2.28 tok/s turn)
#
# THERMAL FLAG: Run D's ~24-minute full-window cold prefill held tj at 106-109 C and raised
#   gpu-throttle-alert 238 times, on the fan's most aggressive stock profile (nvfancontrol
#   `cool`, close_loop). A 1M-context lane at util 0.7 on this box is thermally bounded.
#
# NOT DECIDED HERE (operator decisions): DSpark adoption, fp8 vs bf16 KV, the budget/shape for
# an associate lane on the Thor, and the worker lane's disposition (frame vagueness v2).
#
# ============================== COMPARISON TABLES ==============================
# (generated by compare.py over the three raw result directories; decode at depth shown only
#  for >= 64 forced tokens; the Run C "cold 1M" row is the needle launched as "warm" that was in
#  fact a cold prefill — see the Run C CORRECTION below)


### Probes and single-stream decode

| metric | plain fp8 | DSpark fp8 | plain bf16 |
|---|---|---|---|
| known-answer | PASS | PASS | PASS |
| multi-step | PASS | PASS | PASS |
| tool call | PASS | PASS | PASS |
| prose decode (tok/s, 3 runs) | 79.25, 78.72, 78.73 | 86.69, 77.41, 84.27 | 61.12, 73.79, 73.86 |
| code decode (tok/s) | 78.68 | 141.14 | 73.7 |

### Needles (decode shown only for >= 64 forced tokens)

| metric | plain fp8 | DSpark fp8 | plain bf16 |
|---|---|---|---|
| 128,000 cold: recall / TTFT s | PASS / 46.84 | PASS / 49.91 | PASS / 45.28 |
| 128,000 decode at depth (tok/s) | 70.83 | 67.46 | 61.16 |
| 250,000 cold: recall / TTFT s | PASS / 131.7 | PASS / 128.78 | PASS / 122.44 |
| 250,000 decode at depth (tok/s) | 66.66 | 66.07 | 52.73 |
| 1,040,000 cold: recall / TTFT s | PASS / 1687.85 | PASS / 1661.2 | PASS / 1448.97 |
| 1,040,000 decode at depth (tok/s) | 33.41 | 45.02 | 26.82 |

### Concurrency: aggregate tok/s / per-request p50

| N | plain fp8 | DSpark fp8 | plain bf16 |
|---|---|---|---|
| 1 | 76.2 / 77.86 | 75.0 / 76.64 | 73.3 / 74.61 |
| 2 | 108.5 / 56.68 | 94.8 / 48.95 | 112.2 / 58.25 |
| 4 | 158.3 / 40.07 | 157.7 / 40.96 | 174.2 / 44.08 |
| 8 | 267.1 / 33.75 | 249.3 / 32.4 | 255.4 / 32.26 |
| 16 | 370.7 / 23.49 | 322.9 / 21.51 | 358.3 / 22.71 |
| 32 | 541.0 / 17.17 | 516.6 / 17.02 | 527.8 / 16.7 |

### Agentic (12 turns x ~8K tool-result tokens, to ~118K)

| metric | plain fp8 | DSpark fp8 | plain bf16 |
|---|---|---|---|
| 1 session(s): tool calls | 12/12 | 12/12 | 12/12 |
| 1 session(s): recall | 3/3 | 3/3 | 3/3 |
| 1 session(s): wall s | 54.7 | 65.1 | 47.8 |
| 1 session(s): TTFT p50 s | 3.897 | 5.181 | 3.592 |
| 1 session(s): TTFT max s | 5.485 | 7.552 | 4.9 |
| 1 session(s): decode min tok/s | 88.69 | 202.86 | 81.9 |
| 4 session(s): tool calls | 48/48 | 48/48 | 48/48 |
| 4 session(s): recall | 12/12 | 12/12 | 12/12 |
| 4 session(s): wall s | 153.6 | 184.7 | 138.6 |
| 4 session(s): TTFT p50 s | 11.475 | 11.742 | 9.981 |
| 4 session(s): TTFT max s | 15.978 | 21.777 | 14.814 |
| 4 session(s): decode min tok/s | 51.33 | 2.28 | 33.9 |

# ============================== RUN-BY-RUN RECORD ==============================

# spike — Nemotron 3.5 Lightning SERVES on the Jetson AGX Thor on vLLM v0.29.0 (the 2026-08-20 NO-GO reversed)
#
# Frame: .devague lightning-on-thor (scoped + /think 2026-09-13). Box: Jetson AGX Thor
# (sm_110, 122.82 GiB unified, L4T R38.2, driver 580.00). Operator: lobes (Claude),
# directed session; worker lane stopped for the session with operator approval.
# Measure-only: no shape/role/code change lands with this transcript (#108).
#
# Reverses: docs/evidence/2026-08-20-spike-lightning-thor-no-go.txt (Mamba2 SSD Triton
# warmup wedged 25+ min on nightly 8bd082 / vLLM 0.26.1rc1 and on v0.27.1).

## Engine

image  vllm/vllm-openai:v0.29.0-aarch64
digest vllm/vllm-openai@sha256:18372a7224938643461b846fb64c5c9d3d6e9727e82caf2dc3043e620c9d4d7a
$ docker run --rm --runtime nvidia --entrypoint python3 <image> -c "<torch/vllm/device probe>"
torch 2.13.0+cu130 cuda 13.0
arch ['sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
vllm 0.29.0
dev NVIDIA Thor (11, 0)
matmul ok True

## Run A — 128000, no speculation, util 0.7, kv fp8 (the Jetson AI Lab recipe minus DSpark)

Launcher: bridge network, -p 127.0.0.1:8090:8000 (never --network host), HF cache mounted,
drop_caches before boot. Args: --max-model-len 128000 --kv-cache-dtype fp8
--gpu-memory-utilization 0.7 --max-num-batched-tokens 16384 --enable-prefix-caching
--reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
--trust-remote-code --mamba-backend flashinfer --mamba-ssm-cache-dtype float16
--enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --mamba-cache-mode align

Before boot: Mem 122 total / 25 used / 97 available (worker lane stopped, embed+rerank+prod stack up).

Boot log (excerpts):
  21:41:18 [nvfp4.py:304] Using 'MARLIN' NvFp4 MoE backend
  21:41:18 [cuda.py:492] Using FLASHINFER attention backend
  21:41:36 [model_runner.py:404] Model loading took 17.82 GiB memory and 18.52 seconds
  21:41:52 [mamba_mixer2.py:597] Warming up Mamba2 SSD Triton kernels...
  21:43:23 [monitor.py:81] Initial profiling/warmup run took 91.13 s      <-- the step that wedged on 2026-08-20
  21:43:25 [ssu_dispatch.py:310] Using flashinfer Mamba SSU backend.
  21:44:16 [gpu_worker.py:625] Available KV cache memory: 60.21 GiB
  21:44:16 [kv_cache_utils.py:2032] GPU KV cache size: 18,346,666 tokens, Maximum concurrency for 128,000 tokens per request: 143.33x
           ^ KV-POOL CAPACITY CEILING (KV cache tokens / max_model_len), NOT measured concurrency and NOT a throughput multiplier; measured aggregate throughput is the N=1-32 sweep.
  21:46:32 [gpu_worker.py:860] Free memory on device (94.49/122.82 GiB) on startup ... 22.84 GiB consumed (weights + non-torch), 2.92 GiB peak activation
  21:46:38 [core.py:361] init engine took 301.61 s (compilation: 14.36 s)
  /health 200 at +362 s from docker run

Probes (temperature 0, direct to the container on 127.0.0.1:8090, no proxy):
  known_answer  17+25          -> "42"      PASS  2.05 s
  multi_step    09:40 train    -> "15:40"   PASS  6.38 s
  tool_call     get_weather    -> finish_reason=tool_calls, get_weather({"city": "Tel Aviv", "unit": "c"})  PASS  0.83 s
Decode, 1 request, thinking off:
  prose  31-token prompt, 500/481/479 tokens: 79.25 / 78.72 / 78.73 tok/s, TTFT 0.068-0.072 s
  code   33-token prompt, 768 tokens:          78.68 tok/s, TTFT 0.074 s
Needle (non-streamed): 88,940 prompt tokens, secret at 37% depth -> "MAROON-4817" PASS, 28.2 s wall, ~3,159 tok/s prefill

## Run B — 1,048,576 (native max_position_embeddings), no speculation

Same launcher/args as Run A except --max-model-len 1048576 (no YaRN / hf_overrides needed:
native ceiling). A first launch attempt never started — the watcher's own
`pkill -f "docker logs -f lightning-spike"` matched its ssh shell and killed it (operator
tooling error, not an engine event); the relaunch below is the real Run B.
Before relaunch: free showed 112 GiB used with no spike container and 9.5 GiB total
process RSS; sync + drop_caches returned it to 25 used / 97 available (reclaimable
cache, NOT the driver-leak signature from 2026-08-11).

Boot log (excerpts):
  21:51:42 [nvfp4.py:304] Using 'MARLIN' NvFp4 MoE backend
  21:51:42 [cuda.py:492] Using FLASHINFER attention backend
  21:52:15 [mamba_mixer2.py:597] Warming up Mamba2 SSD Triton kernels...
  21:53:46 [monitor.py:81] Initial profiling/warmup run took 90.83 s
  21:54:37 [model_runner.py:960] Graph capturing finished in 49 secs, took 1.87 GiB
  21:54:38 [gpu_worker.py:625] Available KV cache memory: 59.44 GiB
  21:54:38 [kv_cache_utils.py:2032] GPU KV cache size: 20,431,534 tokens, Maximum concurrency for 1,048,576 tokens per request: 19.49x
           ^ KV-POOL CAPACITY CEILING (KV cache tokens / max_model_len), NOT measured concurrency and NOT a throughput multiplier; measured aggregate throughput is the N=1-32 sweep.
  21:59:52 [core.py:361] init engine took 473.29 s (compilation: 14.82 s)
  /health 200 at +559 s from docker run

Serial measurement chain on this server (one at a time, clean timings):
  needles at ~128K / ~250K / ~1,040,000 prompt tokens (streamed: TTFT, prefill, decode at depth)
  concurrency sweep 1,2,4,8,16,32 x 512 tokens
  agentic: 1 session x 12 turns x ~8K tokens/turn tool results, then 4 concurrent sessions

Needle ~128K (streamed, thinking off, secret at 37% depth):
  prompt 128,073 tokens -> "MAROON-4817" PASS, TTFT 46.84 s, prefill ~2,734.5 tok/s
  decode_tok_s_at_depth 278.8 is INVALID: the model emitted only the code (10 completion
  tokens) and ignored the story instruction, so the rate spans 9 tokens. Decode at depth is
  re-measured after the chain with min_tokens=256 + ignore_eos (probe fixed locally, not
  swapped mid-chain so every needle in this chain ran the same probe version). The same flaw
  applies to the ~250K and ~1,040,000 needles in this chain: their PASS/TTFT/prefill stand,
  their decode figures do not.

Needle ~250K (same probe version):
  prompt 250,073 tokens -> "MAROON-4817" PASS, TTFT 131.7 s, prefill ~1,898.9 tok/s
  decode_tok_s_at_depth 1482.9 INVALID for the same reason (10 completion tokens).

Prefill throughput falls with depth (single request, chunked prefill 16384):
  88,940 -> ~3,159 tok/s | 128,073 -> ~2,735 tok/s | 250,073 -> ~1,899 tok/s

Two false alarms during the 250K needle, recorded so they are not misread later:
  1. free showed 121/122 GiB used, 1.8 GiB available, swap 10.5/59.6 GiB. Total process RSS
     was 8.9 GiB; "used" is the static unified-memory GPU reservation (util 0.7 + embed +
     rerank lanes). vmstat over 20 s: so=0, si 0-72 KB/s, CPU 96-97% idle, wa 0 — no
     thrashing; prod containers healthy. dmesg carries NVRM NV_ERR_NO_MEMORY lines at
     21:44 and 21:57 UTC (inside both boots' profiling/graph-capture windows); both engines
     recovered and served.
  2. The engine's 10 s "Engine 000" stats line went silent from 22:01:49 to 22:03:49 with a
     request running and host CPU idle — the Aug-20 wedge signature. It was NOT a wedge:
     the stats logger does not print mid-prefill of one huge request; it printed once at
     the end (avg prompt throughput 25,005.9 tok/s) and the needle returned PASS.

Thermals: tegrastats during the 250K prefill tail showed tj 81.6-96.7 C (gpu == tj).
tegrastats logged every 5 s to results/tegrastats.log for the rest of the chain.
Throttle check at 22:05:04 UTC (during the ~1,040,000-token prefill), power mode MAXN:
  tj-thermal 97.8 C (active fan trips 80/86/91/100 C, critical 114.5 C)
  gpu-thermal 97.9 C (passive/throttle trip 109 C, critical 114.5 C) -> ~11 C headroom
  cooling devices: devfreq-gpu-gpc-0 / devfreq-gpu-nvd-0 cur_state 0, gpu/cpu/soc throttle-alert 0,
  every cpufreq cur_state 0 -> NO thermal throttling; pwm-fan stage 2/4 (pwm 144).
  Fan policy left untouched (box-wide setting, not needed).

Needle ~1,040,000 (same probe version):
  prompt 1,040,073 tokens -> "MAROON-4817" PASS, TTFT 1,687.85 s (28.1 min), prefill ~616.2 tok/s
  decode figure INVALID (10 completion tokens). Recall at 99.2% of the native window PASSES.

Prefill throughput vs depth (cold, single request, chunked prefill 16384):
     88,940 -> ~3,159 tok/s
    128,073 -> ~2,735 tok/s
    250,073 -> ~1,899 tok/s
  1,040,073 ->   ~616 tok/s   (full-window cold prefill is a ~28 min wait)

## Concurrency sweep (Run B server, 1M window, no speculation)

N simultaneous streamed chats, ~31-token prompts, 512 completion tokens each, thinking off:
  N  | wall s | aggregate tok/s | per-request decode p50 (min) | TTFT p50 / max
   1 |   6.72 |            76.2 | 77.86 (77.86)                | 0.157 / 0.157 s
   2 |   9.44 |           108.5 | 56.68 (56.68)                | 0.397 / 0.419 s
   4 |  12.94 |           158.3 | 40.07 (40.05)                | 0.184 / 0.185 s
   8 |  15.34 |           267.1 | 33.75 (33.73)                | 0.190 / 0.192 s
  16 |  22.10 |           370.7 | 23.49 (23.47)                | 0.339 / 0.343 s
  32 |  30.28 |           541.0 | 17.17 (17.01)                | 0.511 / 0.516 s
Aggregate throughput scales 7.1x from N=1 to N=32 with no knee reached; per-request decode
falls to 22% of single-stream. No request failed.

## Agentic workload (Run B server)

Loop: system prompt + 2 tools (read_file, finish). Each turn the user names a path; the model
must call read_file with that exact path; the tool result injects ~8K tokens of synthetic
source with one "retry budget" fact at a random position. 12 turns, then a final request to
call finish with the budgets from turns 1, 7 and 12 (early / middle / late), temperature 0,
thinking off. Prefix caching is ON, so each turn prefills only the newly appended ~10K tokens.

1 session x 12 turns:
  tool calls valid (correct tool AND exact path) 12/12; recall 3/3 at 117,929 prompt tokens
  per-turn prompt tokens 379 -> 108,105; TTFT 0.32 s -> 5.49 s (p50 3.90 s)
  wall 54.7 s for the whole session
4 concurrent sessions x 12 turns:
  tool calls valid 48/48; recall 12/12 (each session 3/3) at 117,929-118,013 prompt tokens
  TTFT p50 11.48 s, max 15.98 s (final-report turns 1.0-18.5 s); wall 153.6 s
CAVEAT: agentic per-turn decode figures (single 88.7-109.9, concurrent p50 59.2 / min 51.3
tok/s) span only ~29-30 completion tokens per turn — short bursts, not sustained decode.
Use the concurrency sweep for sustained decode.

## Decode at depth (Run B server, fixed probe)

Same deterministic needle prompts re-sent with min_tokens=256 + ignore_eos (256 completion
tokens each). The prompts were already in the prefix cache, so TTFT below is a WARM number,
not cold prefill (cold figures are the needle table above). Recall re-checked on the head:
    prompt   | recall | warm TTFT | decode tok/s at depth
         ~31 |   —    |  ~0.07 s  | 78.7-79.3 (Run A single stream)
     128,073 |  PASS  |   0.95 s  | 70.83
     250,073 |  PASS  |   1.97 s  | 66.66
   1,040,073 |  PASS  |   9.14 s  | 33.41
Decode holds within ~16% of single-stream out to 250K and roughly halves at the full 1M
window — consistent with the minority attention layers' per-token KV reads growing with
context while the Mamba-2 layers carry constant state.

## Thermals and power across the whole chain (tegrastats, 5 s, 398 samples)

Window 22:04:27 -> 22:37:33 UTC (1M needle, concurrency sweep, both agentic runs), MAXN:
  tj          min 75.2 C | p50 98.0 C | p95 98.7 C | max 99.5 C (9 samples >= 99 C)
  VDD_GPU     p50 67.0 W | p95 69.0 W | max 69.8 W | mean 63.1 W
  VIN (board) p50 101.0 W | max 150.1 W
  RAM used    121.4-121.9 GiB (flat) | swap 10.5-10.8 GiB (flat)
GPU passive trip is 109 C: no sample came within 9.5 C. Throttle re-check at 22:38:09 UTC:
every devfreq-gpu / throttle-alert / cpufreq cooling device cur_state 0. No throttling at
any point measured. (Thor tegrastats carries no GR3D_FREQ load field; VDD_GPU is the proxy.)

## FP4 kernel capability on the Thor (GPU quiet, between Run B and Run C)

Both Run A and Run B selected the 'MARLIN' NvFp4 MoE backend and MarlinNvFp4LinearKernel for
NVFP4 GEMM, and logged "Your GPU does not have native support for FP4 computation ... Marlin".
Direct check in the same image on the Thor GPU:
$ docker run --rm --runtime nvidia ... vllm/vllm-openai:v0.29.0-aarch64 python3 /fp4check.py
vllm 0.29.0 torch 2.13.0+cu130 dev NVIDIA Thor (11, 0)
capability to_int 110
cutlass_fp4_supported() True
cutlass_scaled_mm_supports_fp4(100) True
cutlass_scaled_mm_supports_fp4(110) True
cutlass_scaled_mm_supports_fp4(120) True
cutlass_scaled_mm_supports_fp4(121) True
has_flashinfer_cutlass_fused_moe() True
=> the "no native FP4" warning is NOT a statement about sm_110: vLLM's CUTLASS FP4 path and
FlashInfer CUTLASS fused MoE both report support on capability 110.

Why MARLIN anyway — the checkpoint, not the card (snapshot bee7596271d1, safetensors index,
18,487 tensors):
  experts.*.weight           6144
  experts.*.weight_scale     5888
  experts.*.weight_scale_2   5888
  experts.*.input_scale         0   <-- no activation scales on ANY expert projection
  in_proj.*.input_scale        23   (FP8 layers)
  out_proj.*.input_scale       23   (FP8 layers)
The boot log's own detection agrees: "Detected ModelOpt NVFP4 checkpoint (quant_algo=
W4A16_NVFP4)". Native NVFP4 MoE kernels compute in W4A4 and need per-layer activation
scales the experts do not carry, so weight-only Marlin is this checkpoint's MoE path on any
GPU (the Orin and, per the 2026-08-20 transcript's family, the Spark serve it the same way).
No forced --moe-backend arm was booted: it would be refused or run without activation scales.
CONCLUSION: there is no hidden native-FP4 decode speedup to unlock on the Thor for this
checkpoint.

## Run C — DSpark x5, 1,048,576, util 0.7, kv fp8

Same launcher/args as Run B plus the recipe's speculative flags:
  --speculative_config.method dspark
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
  --speculative_config.num_speculative_tokens 5
Drafter: snapshot downloaded 2026-09-13 as root inside the fleet nightly image (the host HF
cache lock dir is root-owned; a host-side `hf download` failed with EACCES on the .locks file).

Boot log (excerpts so far):
  22:39:57 [model.py:684] Resolved architecture: Qwen3DSparkModel
  22:39:57 [arg_utils.py:2404] Hybrid model with EAGLE speculative decoding: defaulting
           prefix_cache_retention_interval to dense checkpointing.
  22:40:30 [model_runner.py:404] Model loading took 19.44 GiB memory (Run B: 17.82 GiB -> drafter ~1.62 GiB)
  22:40:47 [mamba_mixer2.py:597] Warming up Mamba2 SSD Triton kernels...

  22:40:28 [cuda.py:492] Using TRITON_ATTN attention backend (drafter)
  22:42:18 [monitor.py:81] Initial profiling/warmup run took 91.05 s
  22:42:50 [compilation.py:1465] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with
           spec-decode for attention backend FlashInferBackend; setting cudagraph_mode=PIECEWISE
  22:43:32 [gpu_worker.py:625] Available KV cache memory: 56.27 GiB
  22:43:32 [kv_cache_utils.py:2032] GPU KV cache size: 17,911,473 tokens, Maximum concurrency
           for 1,048,576 tokens per request: 17.08x   (Run B: 20,431,534 / 19.49x)
           ^ KV-POOL CAPACITY CEILING (KV cache tokens / max_model_len), NOT measured concurrency and NOT a throughput multiplier; measured aggregate throughput is the N=1-32 sweep.
  22:45:39 [core.py:361] init engine took 309.09 s (compilation: 45.01 s)
  /health 200 at +392 s from docker run

Measurement pass: lightning-phase.sh results-dspark with SKIP_COLD_1M=1 (the cold 1M needle
is prefill-dominated, which speculation does not accelerate, and was already measured in
Run B at 1,687.85 s); warm 1M needle still measures decode at 1M depth.

Operator-tooling incident during the pass (not an engine event): at ~22:47 UTC the Claude Code
harness on the Spark stopped the LOCAL ssh client ("system is running low on memory" — Spark
side: 93/121 GiB used, 28 GiB available, cortex lane resident). The remote phase.sh kept
running but had lost its stdout, so it would have died on SIGPIPE at its next echo. The
remainder was relaunched detached (nohup setsid, output to files, exact-PID stop of the old
runner after its in-flight 250K needle) and completion polled by short ssh checks. Results
already written to disk by the original runner stand.

Probes (DSpark x5, direct to the container):
  known_answer  "42"     PASS 2.00 s
  multi_step    "15:40"  PASS 3.25 s   (Run A: 6.38 s)
  tool_call     get_weather({"city": "Tel Aviv", "unit": "c"}) finish_reason=tool_calls PASS 0.57 s
Decode, 1 request, thinking off (Run A no-spec in brackets):
  prose  455/464/512 tokens: 86.69 / 77.41 / 84.27 tok/s, TTFT 0.086-0.088 s   [79.25 / 78.72 / 78.73]
  code   768 tokens:         141.14 tok/s, TTFT 0.097 s                       [78.68]
  => DSpark x5: code 1.79x; prose ~1.05x (range 0.98-1.10x) — same shape as the Spark cortex
     DSpark adoption (wins code, flat/loses prose), docs/dspark-speculation.md.
Cold needle ~128K:
  128,073 tokens -> "MAROON-4817" PASS, TTFT 49.91 s, prefill ~2,566 tok/s [Run B cold 46.84 s / ~2,735]
  decode at depth 54.9 tok/s over 256 forced tokens [Run B warm 70.83] — DSpark SLOWER at depth
  on this sample; confirm against the warm re-measure below before drawing a conclusion.

Cold needle ~250K:
  250,073 tokens -> "MAROON-4817" PASS, TTFT 128.78 s, prefill ~1,942 tok/s  [Run B cold 131.7 s / ~1,899]
  decode at depth 63.51 tok/s (256 forced tokens)
Warm decode at depth (prompt already prefix-cached on this server):
    prompt  | recall | warm TTFT | DSpark decode | [Run B no-spec]
   128,073  |  PASS  |  2.01 s   | 67.46 tok/s   | [70.83]
   250,073  |  PASS  |  4.08 s   | 66.07 tok/s   | [66.66]
  => at depth, on this story-generation workload, DSpark x5 is at PARITY with plain decode
     (-4.8% at 128K, -0.9% at 250K). The cold-128K decode figure (54.9) is superseded by the
     warm re-measure. Warm TTFT is ~2x Run B's (2.01 vs 0.95 s; 4.08 vs 1.97 s), consistent
     with the "dense checkpointing" prefix-cache retention vLLM defaults to for a hybrid model
     under EAGLE-style speculation (22:39:57 log line above).
CORRECTION (operator error): SKIP_COLD_1M=1 skipped the only 1M prefill on this fresh server,
so the "warm" 1M needle launched at 22:49:59 is in fact a COLD ~1M prefill. It is recorded as
cold (TTFT) plus decode at 1M depth, not as a warm figure.

Needle ~1M (COLD on this server, see correction above), started 22:49:59:
  1,040,073 tokens -> "MAROON-4817" PASS, TTFT 1,661.2 s (27.7 min), prefill ~626 tok/s
  [Run B cold 1,687.85 s / ~616 tok/s — prefill unchanged by speculation, as expected]
  decode at 1M depth 45.02 tok/s over 256 forced tokens  [Run B warm 33.41 -> DSpark +34.7%]
Decode-at-depth summary, DSpark x5 vs plain (tok/s):
    128K  67.46 vs 70.83  (-4.8%)
    250K  66.07 vs 66.66  (-0.9%)
     1M   45.02 vs 33.41  (+34.7%)
  => speculation's benefit on this workload appears only where plain decode is slowest (full
     window); single samples per depth — treat as indicative.

Concurrency sweep (DSpark x5; same essay prompts, 512 tokens, thinking off; Run B plain in brackets):
  N  | wall s | aggregate tok/s   | per-req decode p50 (min) | TTFT p50 / max
   1 |   6.83 |  75.0 [76.2]      | 76.64 (76.64)            | 0.160 / 0.160 s
   2 |  10.81 |  94.8 [108.5]     | 48.95 (48.95)            | 0.367 / 0.431 s
   4 |  12.99 | 157.7 [158.3]     | 40.96 (40.00)            | 0.214 / 0.215 s
   8 |  16.43 | 249.3 [267.1]     | 32.40 (31.53)            | 0.219 / 0.221 s
  16 |  25.37 | 322.9 [370.7]     | 21.51 (20.49)            | 0.430 / 0.433 s
  32 |  31.72 | 516.6 [541.0]     | 17.02 (16.28)            | 0.592 / 0.599 s
  => on prose, DSpark x5 never beats plain under concurrency (-2% .. -13%); no failures.
     Consistent with prose drafts being rejected often and batching already saturating the
     GPU, so verification cost is not repaid. (Note the N=1 row here is a 512-token essay,
     75.0 tok/s, vs the probe's 768-token code at 141.14 — workload, not noise.)

Agentic 1 session x 12 turns x ~8K tool-result tokens (DSpark x5; Run B plain in brackets):
  tool calls valid 12/12 [12/12]; recall 3/3 at 117,929 prompt tokens [3/3]
  TTFT p50 5.18 s, max 7.55 s              [3.90 s, max 5.49 s]  -> +33% p50
  per-turn decode p50 301.5 tok/s, min 202.9 [104.9, min 88.7]   -> ~2.9x on tool-call bursts
  session wall 65.1 s                       [54.7 s]             -> +19% SLOWER end to end
  => each turn prefills ~10K new tokens and emits only ~29 (a tool call), so TTFT dominates
     the session; DSpark accelerates the tiny, highly predictable tool-call JSON ~3x but the
     prefix-cache path under speculation ("dense checkpointing") costs more per turn than the
     decode saves. Decode figures span ~29 tokens per turn: burst rates, not sustained.

Agentic 4 concurrent sessions x 12 turns (DSpark x5; Run B plain in brackets):
  tool calls valid 48/48 [48/48]; recall 12/12 [12/12] at 117,929-118,013 prompt tokens
  TTFT p50 11.74 s, max 21.78 s [11.48 s, max 15.98 s]
  per-turn decode p50 183.4 tok/s, MIN 2.28 tok/s [p50 59.2, min 51.3] — at least one turn
    nearly stalled under 4-way contention with speculation on
  wall 184.7 s [153.6 s] -> +20% SLOWER end to end
Speculative decoding metrics (engine log, 10 s windows, end of pass):
  tool-call bursts: Mean acceptance length 6.00 (= 1 + 5 drafts), per-position acceptance
    1.000 x5 — every draft accepted on structured tool-call JSON
  final windows: 5.89 -> 4.50 -> 3.25, per-position falling to 1.000/0.583/0.500/0.167/0.000
    as free text takes over
Thermals, DSpark pass (tegrastats, 449 samples): tj p50 98.5 C, max 100.2 C, 71 samples
  >= 99 C (Run B: max 99.5 C, 9 samples) — hotter, still ~9 C under the 109 C passive trip.

## Run C verdict (DSpark x5 on this checkpoint, on the Thor)

  wins : code single-stream 1.79x; decode at 1M depth +35%; tool-call JSON bursts ~3x
  ties : cold prefill at every depth; decode at 128K/250K; correctness (all probes, needles,
         48/48 tool calls, 12/12 recall)
  loses: prose single-stream ~1.05x at best; concurrency -2..-13% aggregate; agentic session
         wall time +19% (1 session) / +20% (4 sessions); warm TTFT ~2x; KV pool -12%
         (17.9M vs 20.4M tokens); one near-stall (2.28 tok/s) under 4-way agentic load
  For the `associate` contract (text, non-coding: code_authoring FORBIDDEN), the only large
  win (code) is out of contract and the measured agent-loop workloads run slower. Adoption is
  an operator decision; this transcript records the trade, it does not make it.

## Run D — plain decode, 1,048,576, util 0.7, KV bfloat16

Why: Runs A-C used --kv-cache-dtype fp8 and logged "Using uncalibrated q_scale 1.0 and/or
prob_scale 1.0 with fp8 attention. This may cause accuracy issues" — the same exposure the Thor
cortex avoided with kv_cache_dtype=auto (#109); the Orin serves this checkpoint with bfloat16
KV. Paired with plain decode because Run C showed DSpark slower on associate's in-contract
workloads. Driver fully detached (nohup setsid, files only), launched 23:25:22 UTC.

  23:30:17 [kv_cache_utils.py:2032] GPU KV cache size: 10,392,269 tokens, Maximum concurrency
           for 1,048,576 tokens per request: 9.91x   (Run B fp8: 20,431,534 / 19.49x)
           ^ KV-POOL CAPACITY CEILING (KV cache tokens / max_model_len), NOT measured concurrency and NOT a throughput multiplier; measured aggregate throughput is the N=1-32 sweep.
  /health 200 at +498 s; pass start 23:33:50 (full pass incl. cold 1M needle, so the warm 1M
  figure is genuinely warm this time)
  23:28:19 [monitor.py:81] Initial profiling/warmup run took 91.75 s
  23:30:17 [gpu_worker.py:625] Available KV cache memory: 60.01 GiB
  23:33:31 [core.py:361] init engine took 419.93 s (compilation: 14.45 s)
  `grep -E "q_scale|uncalibrated"` over this boot log: NO match — the uncalibrated-fp8 warning
  logged by Runs A-C is absent with bfloat16 KV.

Probes (bfloat16 KV; Run A fp8 in brackets):
  known_answer  "42"     PASS 1.95 s  [2.05 s]
  multi_step    "15:40"  PASS 8.29 s  [6.38 s]
  tool_call     get_weather({"city": "Tel Aviv", "unit": "c"}) PASS 1.07 s  [0.83 s]
Decode, 1 request, thinking off:
  prose  463/505/512 tokens: 61.12 / 73.79 / 73.86 tok/s, TTFT 0.078-0.091 s  [79.25 / 78.72 / 78.73]
         (first run low — treated as a warm-up outlier; steady state 73.8)
  code   768 tokens: 73.70 tok/s, TTFT 0.082 s                              [78.68]
  => bfloat16 KV costs ~6-7% single-stream decode at short context vs fp8.

Cold needle ~128K (bfloat16 KV; Run B fp8 cold in brackets):
  128,073 tokens -> "MAROON-4817" PASS, TTFT 45.28 s, prefill ~2,829 tok/s  [46.84 s / ~2,735]
  => KV dtype does not slow prefill.
  decode at depth 62.62 tok/s measured immediately after the cold prefill. Not compared with
  Run B's WARM 70.83: Run C showed a cold-sample decode 19% under its own warm re-measure
  (54.9 vs 67.46). The like-for-like comparison is this run's warm 128K figure (61.16, below).

Cold needle ~250K (bfloat16 KV; Run B fp8 cold / Run C DSpark cold in brackets):
  250,073 tokens -> "MAROON-4817" PASS, TTFT 122.44 s, prefill ~2,043 tok/s  [131.7 s / 128.78 s]
  decode immediately after cold prefill 52.69 tok/s — compare against this run's warm 250K
  figure (52.73, below), not against the other runs' warm numbers.
  => prefill at parity or slightly faster than fp8 at 128K and 250K (single samples each).

Cold needle ~1M (bfloat16 KV; Run B fp8 cold / Run C DSpark cold in brackets):
  1,040,073 tokens -> "MAROON-4817" PASS, TTFT 1,448.97 s (24.1 min), prefill ~718 tok/s
  [1,687.85 s / 1,661.2 s] -> ~14% faster cold prefill than fp8 at the full window
  decode immediately after cold prefill 26.94 tok/s (cold sample; see warm figure)
  => across 128K / 250K / 1M, bfloat16-KV cold prefill was at parity or faster than fp8 on
     every sample (single sample per depth); a plausible cause is avoiding fp8 KV
     quantize/dequantize in the attention path during prefill — not verified here.

Warm decode at depth (bfloat16 KV; prompts prefix-cached by the cold pass on THIS server):
    prompt  | recall | warm TTFT | bf16 decode | [Run B fp8 warm] | [Run C DSpark warm]
   128,073  |  PASS  |  0.80 s   | 61.16       | [70.83]          | [67.46]
    250,073  |  PASS  |  1.19 s   | 52.73       | [66.66]          | [66.07]
 1,040,073  |  PASS  |  4.68 s   | 26.82       | [33.41]          | [45.02]
  bf16 vs fp8 decode gap by depth: short context -6.5% (73.8 vs ~79), 128K -13.7%,
  250K -20.9%, 1M -19.7% — consistent with attention reading 2x the KV bytes per token.
  Warm TTFT is faster than fp8's at every depth (0.80 vs 0.95 s; 1.19 vs 1.97 s; 4.68 vs 9.14 s).
  Recall PASS at every depth on bfloat16, as it was on fp8 — this probe set found NO accuracy
  difference attributable to the uncalibrated fp8 KV scales.

Concurrency sweep (bfloat16 KV; same essay prompts, 512 tokens; Run B fp8 / Run C DSpark in brackets):
  N  | wall s | aggregate tok/s        | per-req decode p50 (min) | TTFT p50 / max
   1 |   6.99 |  73.3 [76.2 / 75.0]    | 74.61 (74.61)            | 0.135 / 0.135 s
   2 |   9.13 | 112.2 [108.5 / 94.8]   | 58.25 (58.25)            | 0.355 / 0.393 s
   4 |  11.76 | 174.2 [158.3 / 157.7]  | 44.08 (43.88)            | 0.166 / 0.167 s
   8 |  16.04 | 255.4 [267.1 / 249.3]  | 32.26 (32.11)            | 0.196 / 0.197 s
  16 |  22.86 | 358.3 [370.7 / 322.9]  | 22.71 (22.62)            | 0.356 / 0.360 s
  32 |  31.04 | 527.8 [541.0 / 516.6]  | 16.70 (16.54)            | 0.427 / 0.432 s
  => parity with fp8 within noise (-4% .. +10%), no failures. Expected: ~31-token prompts keep
     the KV cache tiny, so KV dtype barely enters; the dtype cost appears at depth (above).

Agentic 1 session x 12 turns (bfloat16 KV; Run B fp8 / Run C DSpark in brackets):
  tool calls valid 12/12; recall 3/3 at 117,929 prompt tokens
  wall 47.8 s                [54.7 s / 65.1 s]   -> fastest of the three (-13% vs fp8)
  TTFT p50 3.59 s, max 4.90  [3.90, 5.49 / 5.18, 7.55]
  per-turn decode p50 94.65, min 81.9 tok/s  [104.9, 88.7 / 301.5, 202.9] (~29-token bursts)
  => each turn is prefill-dominated (~10K tokens in, ~29 out), so bfloat16's faster prefill
     outweighs its slower decode in this agent loop.

Agentic 4 concurrent sessions x 12 turns (bfloat16 KV; Run B fp8 / Run C DSpark in brackets):
  tool calls valid 48/48; recall 12/12 at 117,929-118,013 prompt tokens
  wall 138.6 s                 [153.6 s / 184.7 s]  -> fastest of the three (-10% vs fp8)
  TTFT p50 9.98 s, max 14.81   [11.48, 15.98 / 11.74, 21.78]
  per-turn decode p50 51.7, min 33.9 tok/s  [59.2, 51.3 / 183.4, 2.28] (~29-token bursts)
Pass DONE 00:06:59 UTC.

### Run D thermals — the GPU hit its 109 C passive trip (READ BEFORE CITING RUN D)

tegrastats (5 s, 397 samples, 23:33:55 -> 00:06:56 UTC): tj p50 106.4 C, max 109.4 C, 326
samples >= 99 C, 16 samples >= 109 C. By 5-minute bucket (tj max / mean, VDD_GPU mean):
  23:30  89.9 /  80.5 C   40.0 W   (boot tail, probes)
  23:35 106.7 / 100.4 C   76.1 W   (cold 128K, cold 250K, cold 1M starts 23:37:30)
  23:40 108.6 / 106.3 C   85.4 W   (cold 1M prefill)
  23:45 109.1 / 107.1 C   85.8 W   (cold 1M prefill)
  23:50 109.4 / 107.1 C   86.0 W   (cold 1M prefill)
  23:55 109.1 / 107.5 C   87.1 W   (cold 1M prefill)
  00:00 109.3 /  96.8 C   59.4 W   (1M prefill ends 00:01:49; warm needles; concurrency)
  00:05 106.1 / 100.9 C   71.7 W   (agentic)
Kernel log: 238 `gpu-throttle-alert cooling state: 0 -> 1` events, first 23:45:30 UTC, last
00:01:33 UTC (Thor dmesg shows local UTC+3: 02:45:30 -> 03:01:33), peaking at 30-32 per minute.
No devfreq / pwm-fan / cpufreq cooling-state change lines were logged.
Measurement windows (result-file mtimes, UTC):
  probes 23:34:33 | cold 128K -23:35:23 | cold 250K -23:37:30  -> BEFORE the first alert: clean
  cold 1M 23:37:30 -> 00:01:49                                  -> ALL 238 alerts fall inside it
  warm 128K/250K/1M -00:02:15 | concurrency -00:03:53 | agentic 1x -00:04:41 | 4x -00:06:59
                                                                -> AFTER the last alert (00:01:33)
Consequences:
  - Run D's cold 1M TTFT (1,448.97 s) was measured UNDER repeated throttle alerts and still beat
    fp8 by 14%; treat it as a conservative figure, not an unthrottled one.
  - Run D's decode-at-depth deficit, concurrency parity and agentic lead were measured after the
    last alert (at tj ~100-106 C), so they are KV-dtype effects, not throttle artefacts.
  - Run D's cold 128K/250K prefill advantage was measured before any alert.
  - Run D drew ~86 W GPU during the 1M prefill vs ~67 W p50 for Run B's whole chain — the bf16
    attention path loads the GPU harder, which matches its faster prefill and explains the heat.
Coverage limit: the dmesg ring's oldest line is 23:39:23 UTC, so it holds no record for Runs A-C.
For those runs the claim "no throttling" rests on tegrastats: tj max 99.5 C (Run B chain) and
100.2 C (Run C pass), both below the 109 C gpu-thermal passive trip that drives
gpu-throttle-alert, plus point checks showing every cooling device at cur_state 0.
OPERATIONAL FLAG: sustained full-window prefill at util 0.7 held tj at 106-109 C for ~25 min.
A 1M-context associate on this box is thermally bounded, not just memory-bounded. Fan state at
idle afterwards: pwm-fan cur_state 0, pwm 75 (fan-control profile check below).

## Fan control (read after Run D, 00:07 UTC, box idle)

$ systemctl is-active nvfancontrol            -> active (since 2026-09-08)
$ grep FAN_DEFAULT /etc/nvfancontrol.conf     -> FAN_DEFAULT_CONTROL close_loop
                                                 FAN_DEFAULT_PROFILE cool
                                                 FAN_DEFAULT_GOVERNOR cont
$ nvfancontrol -q                             -> FAN1:FAN_PROFILE:cool / FAN_GOVERNOR:cont / FAN_CONTROL:close_loop
tj-thermal active trips 80/86/91/100 C map to pwm-fan cdev0-3; nvfancontrol drives the PWM in
close_loop, so the kernel pwm-fan cur_state (read as 0) does not describe the fan. pwm1 at idle
after the run: 67. tj at 00:07:41 UTC: 75.7 C.
=> the fan was already on NVIDIA's most aggressive stock profile. The 106-109 C plateau under
~86 W sustained GPU draw is this box's thermal envelope as cooled today, not a misconfigured
fan. Pinning the fan manually is a box-wide change and was NOT made.

## Operator-tooling incidents (no engine impact)

1. A watcher's `pkill -f "docker logs -f lightning-spike"` matched its own ssh command line and
   killed itself (ssh exit 255) before the first 1M launch; the relaunch is the real Run B.
2. The Claude Code harness on the Spark twice stopped a LOCAL background ssh/poll job for low
   memory (Spark cortex lane resident, ~28 GiB available). Remote work survived; the second
   half of Run C and all of Run D ran fully detached (nohup setsid, file output) with short-poll
   monitoring.
3. The host HF cache lock directory is root-owned; the DSpark drafter was downloaded as root
   inside the fleet nightly image instead.
4. Needle decode figures from the first chain (10-token answers) were invalid and were
   re-measured with min_tokens=256 + ignore_eos; only the re-measured figures are reported.
