SHORT-PREFILL TAX — attribution and the five null controls (2026-09-08)
======================================================================

Question (#2250): a short chat turn's prefill costs far more per token than a
long one. What is the cost, and can it be removed?

INSTRUMENT AND DISCIPLINE
-------------------------
Box 206, one dedicated boot per arm, both 4090s otherwise idle, clocks
unlocked.  Recipe = compose.dflash.yaml (Qwen3.8-27B-exl3-4.0bpw, tkv k4v4,
DFlash K=7, --max-batch 8, --chunk-prefill 2048, GMU 1.0), main-tip source
bind-mounted over the image.

Every number below is the engine's OWN `prefill` stage stamp
(t_first_scheduled_step -> t_first_token) read from
GET /v1/admin/request_timeline, p50 over >=9 requests.  Every request carries
a unique random prefix and `tools/short_prefill/sweep.py` REFUSES any row whose
`cached_tokens` is non-zero, so no row can be a radix-cache hit.  Client wall
clock is reported beside it only as a sanity check.

Instruments used, and what each is good for:
  * request_timeline stage stamps  — the served number.
  * ARBI_STEP_PHASE_PROBE=1        — host phase split, two perf_counter reads
                                     per mark, no CUPTI.
  * py-spy (nsenter into the container, --rate 300, 15651 samples, 8 errors)
                                   — Python attribution of the forward thread.
  * nvidia-smi utilization/power   — coarse GPU-busy and power.
  * --prefill-capture=full         — the null control that removes ALL host
                                     launch cost, so the GPU floor is visible
                                     on its own.

NOT used for attribution: torch.profiler.  It perturbs this engine 2.1x
(TTFT 0.078 -> 0.165 s) and its live start/stop path fails with
"External init callback must run in same thread as registerClient" because the
forward runs on the executor thread; no trace is written.  py-spy also cannot
separate "executing Python" from "blocked inside a C call made from that
Python line" — which is why the capture arm, not py-spy, is the load-bearing
control here.

THE CURVE (eager, the shipped default)
--------------------------------------
  prompt tok   prefill p50   ms per 1k
          41        77 ms       1878
          77        78 ms       1013
         120        76 ms        633
         129        78 ms        605
         152        78 ms        513
         187        77 ms        412
         193        78 ms        404
         255        79 ms        310
         266        85 ms        320
         316        86 ms        272
         389       109 ms        280
         503       113 ms        225
         596       126 ms        211
        1257       238 ms        189
        2724       513 ms        188

On a quiet box the shape is not a decaying efficiency curve: it is DEAD FLAT
at 77-79 ms from 41 to 255 tokens, then steps, then joins a 0.188 ms/token
line with ~0 intercept.  It is `prefill ~= max(floor, 0.188 * N)`.  The
"excess decays" reading in the issue is that max() being progressively hidden,
not a cost that shrinks.  #2250's original "flat 82 ms floor" reading was
right; the 105 numbers that looked like a decay were taken on a loaded box.

WHAT THE FLOOR IS: TWO NEARLY EQUAL COSTS, STACKED
--------------------------------------------------
1. HOST.  The eager prefill forward drives every one of its kernel launches
   from Python (the nsys profile behind ``prefill_cudagraph_buckets`` counts
   ~1500 kernels per forward) and costs ~73-77 ms of host time, independent of
   prompt length.
   ARBI_STEP_PHASE_PROBE puts 72.5 ms of a 193-token step inside
   forward+sample+draft and 3.4 ms blocked on the GPU after the last launch.
   py-spy self-time on the forward thread: 36.8% dynamo/inductor/triton launch
   machinery, 22.6% arbi_serve python, 7.7% torch op dispatch, 5.3% FLA/GDN
   python.  Largest inclusive subtrees: GDN/FLA chain 25 ms, exl3 linear GEMM
   op 14.6 ms (400 linears), tkv attention 4.5 ms.

2. GPU.  With --prefill-capture=full the whole forward is ONE cudaGraphLaunch,
   so the host cost is gone and the served number IS the GPU cost:

     bucket      64    128    192    256    384    512
     prefill   56 ms  58 ms  71 ms  72 ms  87 ms  112 ms

   GPU(N) ~= 35-42 ms fixed + ~15 ms per 128-row int8 GEMM tile.  Buckets 192
   and 256 cost the same, and 64 and 128 cost the same: that is the int8
   prefill leg's TILE_M=128 (shape111) padding rows up to a multiple of 128.

At 193 tokens the two are 77 ms and 71 ms — within 10% of each other.  That is
why every single-sided intervention below moves the number by under 10%.

THE FIVE NULL CONTROLS
----------------------
a) int8 GEMM tile.  Repinned the PREFILL class from shape111 (TILE_M 128) to
   shape85 (TILE_M 64) for every geometry — 1.2-1.55x faster at M=192 in
   tools/int8_gemm/bench_group_scale.py.  Served: 193 -> 76 ms (was 77), and
   2688 -> 604 ms (was 513, +18%).  A kernel that much faster moving 1 ms at
   the short end means the GEMM is not the binding constraint there.

b) Prefill cudagraph capture, full ladder (256/512/1024/2048).  Removes ALL
   host launch cost.  193 -> 72 ms (-6.5%).  But 322 -> 109 (+25%), 610 -> 184
   (+46%), 1259 -> 351 (+47%) because a prompt runs its bucket's full width,
   and servable context falls 194560 -> 144384 tokens (760 -> 564 KV pages,
   -26%).  Rejected.

c) Speculative drafter width.  --mtp-n-draft 7 -> 1: 193 -> 78 ms (was 77).
   DFlash drafts a block, so K does not multiply forwards.

d) Speculation entirely off (no --enable-mtp / --dflash-draft-path), eager:
   39 -> 73, 196 -> 73, 613 -> 120, 2722 -> 501.  The seed-draft chain that
   runs BEFORE the first token is emitted costs ~4 ms of TTFT, not the ~20 ms
   a serial K-step chain would.

e) Prompt-shape churn.  Every request forced to exactly 193 prompt tokens vs
   lengths varying 183-200: 76.5 vs 77.0 ms.  No Dynamo recompile cost, no
   guard thrash.

WHAT IT WOULD TAKE
------------------
Both halves have to come down together.  The capture arm's own bucket-64
number, 56 ms, is what a 41-token prefill would cost with the host half gone;
the eager arm pays 77 ms for the same work.  Removing the host half alone buys
6.5% at 193 tokens because the GPU half then binds; removing the GPU padding
alone buys 1% because the host half then binds.

The only lever that removes the host half is capture, and its price is
measured above: 26% of servable context and a 25-47% regression on the
322-1259-token band that a bucket ladder pads.  A single narrow bucket avoids
the padding regression (prompts above it stay eager) — see the last table in
this file for what that costs and buys.

A CORRECTION TO THE RECORD
--------------------------
config_server.py's prefill_capture note argues from an nsys figure of "~5.7 ms
of launch overhead across ~1500 kernels per forward" and concludes capture
cannot matter.  That figure is the cudaLaunchKernel API time only.  The host
work that PRODUCES those launches — Dynamo entry, the Inductor-generated
launch wrappers, Triton's launcher, the custom-op dispatch, and arbi_serve's
own per-linear Python — is ~73-77 ms per forward, and it is the binding
constraint below ~300 prompt tokens.  The note's conclusion (keep eager) still
holds on this evidence, but for the reason in the table above (bucket padding
and KV pages), not because the host cost is small.

TWO DEFECTS FOUND WHILE MEASURING
---------------------------------
1. FIXED HERE.  `--prefill-capture=full` could not boot at all on a host with
   enough RAM to grant the savepoint ring: `attach_landing` books pinned host
   under the owner name "savepoint.fold_emit_landing", which was never declared
   in PINNED_HOST_OWNERS, so `reserve_pinned_host` refused it by name and the
   boot died in `_profile_and_size_kv`.  That is why #2250's own ask — "measure
   the short-prompt floor under eager vs piecewise/full" — could not be
   answered before now.  One declaration; every capture number in this file is
   from a boot that needed it.

2. OPEN, not fixed here.  A `prefill_cudagraph_buckets` ladder whose rungs are
   far apart mis-selects.  With `ARBI_PREFILL_BUCKETS=256` the auto-extend
   correctly adds 2048 (it logs the warning), but a 506-token prompt then
   replays the 256-token graph and the step dies with
   `RuntimeError: The size of tensor a (256) must match the size of tensor b
   (506) at non-singleton dimension 0`.  With the default ladder
   (256/512/1024/2048) the gaps are small enough that this never shows.  It
   blocks the one capture recipe that would be attractive — a single narrow
   bucket, so short prompts replay and everything above stays eager and
   unregressed.  Worth its own issue.

   Note the VRAM price of that recipe is not small either: a single 256 bucket
   still cost 760 -> 587 KV pages (194560 -> 150272 servable tokens, -23%),
   because the reserve is the capture mempool, not the one graph.
