THE ROW-CLASS LADDER, AND WHAT DE-SPECIALIZING THE SPLIT COSTS — 2026-09-10
===========================================================================

TWO QUESTIONS, ONE RIG.

  1. The boot ladder that warms the row-class-mixed decoder frame drove
     thirteen rungs. How many of them compile anything, and what is the rest
     of the ladder worth?
  2. arbi-serve#2357 makes the ``(verify rows, prefill rows)`` counts reach
     the compiled layer as tensor LENGTHS rather than Python ints. It was
     measured slower at c=64 and not shipped, and the cost was never
     isolated. Where does it go?

RIG
  dev-gpu GPU 0 (one 4090), compose.dflash.yaml + compose.bindmount.yaml,
  registry.arbi.work/arbi-serve:test-latest, /models/Qwen3.8-27B-exl3-4.0bpw
  with the DFlash2 6.0bpw drafter (K=7), tkv k4v4, max_batch 8.
  Own compose project, own port, own budget cache seeded from the shared one.
  BASE = main @ a8c2c818b. HEAD = BASE + #2357's three commits.
  Bench: /v1/completions streamed, 28-word prompt, 64 output tokens, decode
  window measured client-side. Two regimes, because they are not the same
  workload: ``cache: false`` with a UNIQUE prefix per request (every request
  prefills), and the identical-prompt/cache-on regime #2357 measured in.

1. WHICH RUNGS COMPILE — FOUR, AND ALWAYS THE FIRST FOUR
   The ladder's receipt now counts Dynamo frames per rung. Thirteen rungs,
   two boots:

     boot 1   13 rungs, 5.88 s, 4 frames, compiled at (1,1) (1,2) (2,1) (2,2)
     boot 2   13 rungs, 6.16 s, 4 frames, compiled at (1,1) (1,2) (2,1) (2,2)

   TORCH_LOGS=recompiles names each step, and the sequence is the mechanism:

     (1,1)  establishes the frame; both counts constant.
     (1,2)  fails ``batch_meta.state_indices`` size 2 -> 3: the row count is
            promoted to a symbol.
     (2,1)  fails ``batch_meta.row_class_split[0] == 1``: the verify count is
            promoted, and Dynamo derives the relation every example so far
            satisfies — ``state_indices.size()[0] == 1 + row_class_split[0]``,
            asked at ``gdn_prefill_conv_flat`` through
            ``check_contiguous_sizes_strides``. That graph is general in the
            verify count and PINNED TO ONE PREFILL ROW.
     (2,2)  fails that relation and refutes it. The two counts are now
            independent, and no later rung can fail a guard.

   The nine rungs after the corners paid for a forward and nothing else.

2. WHAT THE CUT IS WORTH — ~0.3 s, NOT ~0.8 s
   Same code, only the rung list changed (four corners + three checks
   (6,1) (1,6) (4,4)). Two batches of boots. The first pair ran the arms
   back to back; the second INTERLEAVED them one boot at a time, which is
   the one that resolves a difference this size, because the ladder's own
   boot-to-boot spread (~0.3 s) is as large as the effect and drifts with
   what else the box is running:

     back to back   13 rungs   5.88  6.16
                     7 rungs   5.69  5.94  5.80
     interleaved    13 rungs   6.2   6.1   6.1        mean 6.13 s
                     7 rungs   5.5   5.8   5.8        mean 5.70 s

   Interleaved: 0.43 s; on the precompile-HIT boots alone (the L7 first boot
   was a store miss) 6.10 -> 5.80, i.e. 0.30 s. Pooled over all eleven boots,
   6.09 -> 5.76 = 0.33 s, and the two sets do not overlap: every 13-rung boot
   is >= 5.88 s and every 7-rung boot is <= 5.94 s. Call it ~0.3 s, i.e.
   ~50 ms per dropped rung — one 128-token prefill row plus the verify rows
   through the stack. The four compiles are still the ladder's cost; the rung
   count is not. The cut is still worth making: it is free, the
   three checks now have a JOB (a fifth frame at boot is a graph that is not
   general, named by the receipt, instead of a served step discovering it),
   and the receipt says which rungs paid.

3. GENERALITY IS ALREADY HAD ON MAIN
   #2357's headline prize was "one graph for every composition, and the
   3+-prefill-row case served compiled rather than eager". Both are already
   true of main. Driving two long generations against a stream of short
   arrivals until the scheduler admits several prefilling rows in one step:

     fused mixed step FIRED #1: 1 verify rows (8 flat) + 7 prefill rows (1792 flat)
     refusals 0 · post-ready compiles 0        (13-rung and 7-rung ladders alike)

   HEAD compiles the same FOUR frames at the same four rungs. The graph set
   is four either way; what #2357 changes is which guards those four carry.

4. WHERE #2357's COST GOES — DEVICE TIME, IN THE MIXED FORWARD
   ARBI_DEBUG_CAPTURE_LOOKUP=1 times every forward with a CUDA event pair on
   the stream, keyed by (class, B, S). Same bench window on both arms,
   count-matched per shape (the smaller arm's count weights both):

     class          shapes  calls    base us/call  head us/call   delta
     replay_exact        8   4788          28,333        28,341   +0.03%
     fused_mixed        19    959         107,149       110,212   +2.86%

   ``replay_exact`` is the captured decode/verify graph, which the change
   does not touch: flat to +0.03% across eight shapes, which is what the
   instrument resolves. ``fused_mixed`` is the row-class-mixed forward, which
   is the only path the change touches, and it is where all of the cost is.
   It is not uniform — the narrow mixed forwards carry it and the widest
   barely do:

     B=8,S=368   100,708 -> 106,655   +5.90%
     B=8,S=371   102,809 -> 105,800   +2.91%
     B=8,S=370   103,944 -> 104,986   +1.00%
     B=8,S=676   145,662 -> 147,693   +1.39%
     B=8,S=677   146,475 -> 146,284   -0.13%

   HOST IS RULED OUT. The step-phase probe puts +1.83 ms on the mixed step's
   metadata-build phase (``fused_halves_built -> fused_batch_built``, 9.45 ->
   11.28 ms) and +2.3 ms on the forward phase, while the untouched pairs move
   under 1% (``step_begin -> sched_done`` 2415 -> 2427 us). But the ONE
   function the change adds to that phase costs, priced in the image on CPU:

     _attach_row_class_split   5,916 ns/call (BASE)   6,427 ns/call (HEAD)

   0.51 us against a 1,830 us phase move. The host-side rise is downstream of
   a busier device, not the change's own host work — which is also why
   memoizing the two moved ``arange``s per composition (#2357's third commit,
   present in HEAD here) recovered nothing.

   AND IT IS NOT MORE KERNELS. TORCH_LOGS=output_code on a boot per arm, over
   the four mixed frames the ladder compiles (they warm OUTSIDE the
   precompile window, so they compile on every boot and their code is always
   printed):

     arm    triton kernel defs   launches in call()   device allocs
     BASE                   22                   22              20
     HEAD                   21                   21              19

   HEAD issues one kernel FEWER and one allocation fewer — the ``arange``
   Inductor no longer has to emit, which is what #2357's third commit moved
   out of the frame. It is slower anyway. So the cost is not a lost fusion,
   not an extra copy and not an added launch: it is in the KERNELS
   THEMSELVES, which is what a symbolic extent buys where a constant used to
   be — grid and block sizes that can no longer be folded at compile time.
   That also fits the shape curve above, where the narrow mixed forwards pay
   most and the widest barely pay at all.

5. WHAT THAT IS WORTH END TO END
   At c=64 the mixed forward is 55% of all timed device time (178.6 s of
   324.1 s in the window), and it is where the prefill chunks run. So the
   effect lands on latency first and on the decode-only rate least:

   identical-prompt / cache-on regime (n=4 reps, 95% CI):
     median total s     BASE 6.902 [6.832, 6.972]   HEAD 7.418 [7.304, 7.531]   +7.5%
     median TTFT ms     BASE 5657  [5579,  5736 ]   HEAD 6039  [5952,  6126 ]   +6.7%
     per-request TPOT-1 BASE 20.54 [20.28, 20.80]   HEAD 20.28 [20.18, 20.39]   -1.2%

   cache-off / unique-prefix regime (n=6 reps):
     c=64 TPOT-1        BASE 14.57 [14.43, 14.71]   HEAD 14.45 [14.35, 14.55]   -0.8%
     c=8  TPOT-1        BASE 21.23 [20.97, 21.49]   HEAD 21.55 [21.35, 21.75]   +1.5%
     c=1  TPOT-1        BASE 39.23 [39.11, 39.35]   HEAD 39.13 [39.02, 39.23]   -0.3%

   The two latency intervals are disjoint; the throughput ones are not, which
   is why the device-time table above is the evidence and the throughput
   table is the consequence.

   Acceptance was equal on the memorised regime (1.0 both arms, identical
   output hash) and NOT equal on free prose (BASE 0.3602, HEAD 0.3207, and
   the two arms' completions hash differently), so the prose throughput
   figures carry an acceptance difference as well as a device one. The
   latency figures above are from the fixed-prompt bench, where both arms
   emit the same text.

6. WHAT THE FOUR CORNERS DO *NOT* REST ON
   The reading that (1,n) and (n,1) each hold their own graph because Dynamo
   specializes a length-0/1 dimension unconditionally is a real rule, and it
   is what #2357's own commit message gives as the mechanism. It is NOT what
   this boot's recompile log attributes MAIN's four traces to, and the
   distinction is worth keeping because the two make the same prediction here
   and would not everywhere.

   On main the counts arrive as Python ints, and the log names an ordinary
   promotion sequence: a size mismatch 2 -> 3 on ``state_indices``, then an
   int-value guard ``row_class_split[0] == 1``, then a DERIVED relation
   ``state_indices.size()[0] == 1 + row_class_split[0]`` that (2,2) refutes.
   The only 0/1 note anywhere in the log sits on a different frame boundary
   entirely — ``2 <= batch_meta.state_indices.size()[0]``, the B==1
   decode-vs-multirow split, flagged as "0/1 specialization in the framework".

   So the cut is not safe BECAUSE of the 0/1 rule. It is safe because the
   outcome was driven and observed: with the cut ladder the scheduler was
   made to admit a 7-prefill-row mixed step and it served COMPILED, with zero
   serving recompile refusals and zero post-ready compiles (section 3), and
   the boot's own receipt now names any rung that compiles outside the four.

VERDICT
  The ladder cut lands: measured, free, and it makes the ladder's own claim
  checkable. #2357 stays closed — the generality it was for is already had
  on main, and it pays for a shorter boot ladder with device time on the
  hottest mixed path.
