ptxas register/spill census: exl3_gemm_kernel, ordered chain vs stream-K parallel fixup
==========================================================================================

Built in registry.arbi.work/arbi-serve:test-latest, CUDA 13.0, -arch=sm_89, over
quant/comp_units/exl3_comp_unit_{1..8}_cb0.cu -- bit widths 1-8, codebook 0, both
C dtypes. Baseline arbi-dev/exllamav3 @ 7b428549; arm perf/gemm-parallel-fixup.

WHY. The fixup adds a second place a C-tile leaves and re-enters registers, in an
epilogue several shipped instantiations already sit at the 128-register wall in.
A register count is a PREDICTOR of time, never a measurement of it. This is a risk
flag to be settled on a card, and the kernel is default-off until it is.

THE GATE. `fixup_shape_ok` is compile-time: two staging slots per CTA at a grid
bound of 256 must fit the 16 MiB arena, i.e. TILESIZE_M * TILESIZE_N <= 8192. An
ineligible instantiation emits none of the staging path and none of its address
arithmetic; the `off` rows below are the check on that claim, not an assertion of
it, and they do not fully pass. 94 of 96 gated instantiations are identical in
both columns. Two (32x512 at bit widths 1 and 3, fp16 C) move by +2 registers
with no spill and no change of occupancy bucket -- the residue of a `bool` that
constant-folds to false but still perturbs ptxas scheduling. Reported rather than
rounded away, because "the gate leaves the rest of the tree untouched" is the
claim the gate exists to make. An earlier draft of this file asserted it and had
a counter-example three lines below.

WHAT THE ADDRESSING COSTS -- three designs measured, not guessed:

  staging addressed like C (r * size_n + n * 8 + c per fragment element)
      +40 to +60 registers, four served shapes newly spilling
  one shared routine for C and staging, base/stride parameters
      worse -- the runtime stride defeats the address folding that made the
      single-call-site version cheap, and it regressed shapes the fixup cannot use
  flat thread-contiguous dump (the layout threadblock_reduce already uses)
      what shipped: a staging slot is private to the fixup, so it carries no layout
      obligation. One pointer and a `+= 4`.

FULL TABLE. `gate` ON means the instantiation carries the staging path.

  TM   TN  TK bits f32  gate |   regs base->arm |   spill base->arm
  16  128  16    1   0    ON |     122 -> 128    |        0 -> 0     
  16  128  16    1   1    ON |     122 -> 128    |        0 -> 0     
  16  128  16    2   0    ON |     123 -> 128    |        0 -> 0     
  16  128  16    2   1    ON |     123 -> 128    |        0 -> 0     
  16  128  16    3   0    ON |     127 -> 128    |        0 -> 0     
  16  128  16    3   1    ON |     127 -> 128    |        0 -> 0     
  16  128  16    4   0    ON |     123 -> 128    |        0 -> 0     
  16  128  16    4   1    ON |     123 -> 128    |        0 -> 0     
  16  128  16    5   0    ON |     128 -> 158    |        0 -> 0     
  16  128  16    5   1    ON |     126 -> 158    |        0 -> 0     
  16  128  16    6   0    ON |     128 -> 158    |        0 -> 0     
  16  128  16    6   1    ON |     126 -> 158    |        0 -> 0     
  16  128  16    7   0    ON |     128 -> 166    |        4 -> 0     
  16  128  16    7   1    ON |     128 -> 166    |        4 -> 0     
  16  128  16    8   0    ON |     127 -> 128    |        0 -> 0     
  16  128  16    8   1    ON |     127 -> 128    |        0 -> 0     
  16  128  32    1   0    ON |     114 -> 126    |        0 -> 0     
  16  128  32    1   1    ON |     114 -> 126    |        0 -> 0     
  16  128  32    2   0    ON |     118 -> 126    |        0 -> 0     
  16  128  32    2   1    ON |     118 -> 126    |        0 -> 0     
  16  128  32    3   0    ON |     115 -> 126    |        0 -> 0     
  16  128  32    3   1    ON |     115 -> 126    |        0 -> 0     
  16  128  32    4   0    ON |     110 -> 126    |        0 -> 0     
  16  128  32    4   1    ON |     110 -> 126    |        0 -> 0     
  16  128  32    5   0    ON |     122 -> 128    |        0 -> 0     
  16  128  32    5   1    ON |     122 -> 127    |        0 -> 0     
  16  128  32    6   0    ON |     122 -> 128    |        0 -> 0     
  16  128  32    6   1    ON |     122 -> 127    |        0 -> 0     
  16  128  32    7   0    ON |     126 -> 128    |        0 -> 24      spill+
  16  128  32    7   1    ON |     126 -> 128    |        0 -> 24      spill+
  16  128  32    8   0    ON |     106 -> 128    |        0 -> 0     
  16  128  32    8   1    ON |     106 -> 128    |        0 -> 0     
  16  256  32    1   0    ON |     128 -> 128    |       32 -> 36      spill+
  16  256  32    1   1    ON |     128 -> 128    |       32 -> 36      spill+
  16  256  32    2   0    ON |     128 -> 128    |       28 -> 44      spill+
  16  256  32    2   1    ON |     128 -> 128    |       32 -> 44      spill+
  16  256  32    3   0    ON |     128 -> 128    |       48 -> 48    
  16  256  32    3   1    ON |     128 -> 128    |       44 -> 48      spill+
  16  256  32    4   0    ON |     128 -> 128    |        0 -> 28      spill+
  16  256  32    4   1    ON |     127 -> 128    |        0 -> 28      spill+
  16  256  32    5   0    ON |     128 -> 128    |       48 -> 56      spill+
  16  256  32    5   1    ON |     128 -> 128    |       48 -> 56      spill+
  16  256  32    6   0    ON |     128 -> 128    |       48 -> 56      spill+
  16  256  32    6   1    ON |     128 -> 128    |       48 -> 56      spill+
  16  256  32    7   0    ON |     128 -> 128    |       80 -> 96      spill+
  16  256  32    7   1    ON |     128 -> 128    |       72 -> 96      spill+
  16  256  32    8   0    ON |     128 -> 128    |       16 -> 20      spill+
  16  256  32    8   1    ON |     128 -> 128    |        0 -> 16      spill+
  16  512  16    1   0    ON |     174 -> 184    |        0 -> 0     
  16  512  16    1   1    ON |     170 -> 174    |        0 -> 0     
  16  512  16    2   0    ON |     180 -> 184    |        0 -> 0     
  16  512  16    2   1    ON |     174 -> 174    |        0 -> 0     
  16  512  16    3   0    ON |     180 -> 198    |        0 -> 0     
  16  512  16    3   1    ON |     180 -> 200    |        0 -> 0     
  16  512  16    4   0    ON |     176 -> 180    |        0 -> 0     
  16  512  16    4   1    ON |     170 -> 175    |        0 -> 0     
  16  512  16    5   0    ON |     196 -> 206    |        0 -> 0     
  16  512  16    5   1    ON |     206 -> 210    |        0 -> 0     
  16  512  16    6   0    ON |     196 -> 206    |        0 -> 0     
  16  512  16    6   1    ON |     206 -> 210    |        0 -> 0     
  16  512  16    7   0    ON |     217 -> 221    |        0 -> 0     
  16  512  16    7   1    ON |     217 -> 222    |        0 -> 0     
  16  512  16    8   0    ON |     177 -> 198    |        0 -> 0     
  16  512  16    8   1    ON |     177 -> 198    |        0 -> 0     
  32  128  16    1   0    ON |     126 -> 160    |        0 -> 0     
  32  128  16    1   1    ON |     128 -> 162    |        0 -> 0     
  32  128  16    2   0    ON |     126 -> 156    |        0 -> 0     
  32  128  16    2   1    ON |     128 -> 158    |        0 -> 0     
  32  128  16    3   0    ON |     128 -> 166    |       32 -> 0     
  32  128  16    3   1    ON |     128 -> 166    |       48 -> 0     
  32  128  16    4   0    ON |     126 -> 160    |        0 -> 0     
  32  128  16    4   1    ON |     128 -> 160    |        0 -> 0     
  32  128  16    5   0    ON |     128 -> 166    |       40 -> 0     
  32  128  16    5   1    ON |     160 -> 166    |        0 -> 0     
  32  128  16    6   0    ON |     128 -> 166    |       40 -> 0     
  32  128  16    6   1    ON |     160 -> 166    |        0 -> 0     
  32  128  16    7   0    ON |     161 -> 168    |        0 -> 0     
  32  128  16    7   1    ON |     166 -> 170    |        0 -> 0     
  32  128  16    8   0    ON |     128 -> 158    |        0 -> 0     
  32  128  16    8   1    ON |     128 -> 160    |       16 -> 0     
  32  128  32    1   0    ON |     128 -> 128    |        0 -> 28      spill+
  32  128  32    1   1    ON |     128 -> 128    |        4 -> 36      spill+
  32  128  32    2   0    ON |     128 -> 128    |        0 -> 28      spill+
  32  128  32    2   1    ON |     128 -> 128    |       16 -> 36      spill+
  32  128  32    3   0    ON |     128 -> 128    |       36 -> 40      spill+
  32  128  32    3   1    ON |     128 -> 128    |       32 -> 88      spill+
  32  128  32    4   0    ON |     128 -> 128    |        0 -> 24      spill+
  32  128  32    4   1    ON |     128 -> 128    |       16 -> 36      spill+
  32  128  32    5   0    ON |     128 -> 128    |       40 -> 48      spill+
  32  128  32    5   1    ON |     128 -> 128    |       48 -> 80      spill+
  32  128  32    6   0    ON |     128 -> 128    |       40 -> 48      spill+
  32  128  32    6   1    ON |     128 -> 128    |       48 -> 80      spill+
  32  128  32    7   0    ON |     128 -> 128    |       60 -> 116     spill+
  32  128  32    7   1    ON |     128 -> 128    |       72 -> 156     spill+
  32  128  32    8   0    ON |     127 -> 128    |        0 -> 16      spill+
  32  128  32    8   1    ON |     128 -> 128    |        0 -> 28      spill+
  32  256  16    1   0    ON |     171 -> 176    |        0 -> 0     
  32  256  16    1   1    ON |     166 -> 172    |        0 -> 0     
  32  256  16    2   0    ON |     172 -> 180    |        0 -> 0     
  32  256  16    2   1    ON |     168 -> 170    |        0 -> 0     
  32  256  16    3   0    ON |     182 -> 196    |        0 -> 0     
  32  256  16    3   1    ON |     176 -> 202    |        0 -> 0     
  32  256  16    4   0    ON |     174 -> 182    |        0 -> 0     
  32  256  16    4   1    ON |     168 -> 170    |        0 -> 0     
  32  256  16    5   0    ON |     186 -> 198    |        0 -> 0     
  32  256  16    5   1    ON |     200 -> 204    |        0 -> 0     
  32  256  16    6   0    ON |     186 -> 198    |        0 -> 0     
  32  256  16    6   1    ON |     200 -> 204    |        0 -> 0     
  32  256  16    7   0    ON |     199 -> 210    |        0 -> 0     
  32  256  16    7   1    ON |     212 -> 214    |        0 -> 0     
  32  256  16    8   0    ON |     171 -> 178    |        0 -> 0     
  32  256  16    8   1    ON |     173 -> 187    |        0 -> 0     
  32  512  16    1   0   off |     242 -> 244    |        0 -> 0       !! gated but moved
  32  512  16    1   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    2   0   off |     244 -> 244    |        0 -> 0     
  32  512  16    2   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    3   0   off |     248 -> 250    |        0 -> 0       !! gated but moved
  32  512  16    3   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    4   0   off |     242 -> 242    |        0 -> 0     
  32  512  16    4   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    5   0   off |     252 -> 252    |        0 -> 0     
  32  512  16    5   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    6   0   off |     252 -> 252    |        0 -> 0     
  32  512  16    6   1   off |     252 -> 252    |        0 -> 0     
  32  512  16    7   0   off |     254 -> 254    |        0 -> 0     
  32  512  16    7   1   off |     255 -> 255    |        0 -> 0     
  32  512  16    8   0   off |     242 -> 242    |        0 -> 0     
  32  512  16    8   1   off |     254 -> 254    |        0 -> 0     
  48  128  16    1   0    ON |     172 -> 186    |        0 -> 0     
  48  128  16    1   1    ON |     186 -> 182    |        0 -> 0     
  48  128  16    2   0    ON |     174 -> 180    |        0 -> 0     
  48  128  16    2   1    ON |     186 -> 184    |        0 -> 0     
  48  128  16    3   0    ON |     180 -> 178    |        0 -> 0     
  48  128  16    3   1    ON |     174 -> 176    |        0 -> 0     
  48  128  16    4   0    ON |     172 -> 180    |        0 -> 0     
  48  128  16    4   1    ON |     184 -> 184    |        0 -> 0     
  48  128  16    5   0    ON |     176 -> 200    |        0 -> 0     
  48  128  16    5   1    ON |     176 -> 208    |        0 -> 0     
  48  128  16    6   0    ON |     182 -> 176    |        0 -> 0     
  48  128  16    6   1    ON |     178 -> 208    |        0 -> 0     
  48  128  16    7   0    ON |     202 -> 208    |        0 -> 0     
  48  128  16    7   1    ON |     214 -> 222    |        0 -> 0     
  48  128  16    8   0    ON |     174 -> 184    |        0 -> 0     
  48  128  16    8   1    ON |     186 -> 182    |        0 -> 0     
  64  128  16    1   0    ON |     184 -> 214    |        0 -> 0     
  64  128  16    1   1    ON |     188 -> 216    |        0 -> 0     
  64  128  16    2   0    ON |     187 -> 212    |        0 -> 0     
  64  128  16    2   1    ON |     190 -> 216    |        0 -> 0     
  64  128  16    3   0    ON |     218 -> 224    |        0 -> 0     
  64  128  16    3   1    ON |     222 -> 228    |        0 -> 0     
  64  128  16    4   0    ON |     176 -> 214    |        0 -> 0     
  64  128  16    4   1    ON |     180 -> 214    |        0 -> 0     
  64  128  16    5   0    ON |     224 -> 228    |        0 -> 0     
  64  128  16    5   1    ON |     228 -> 232    |        0 -> 0     
  64  128  16    6   0    ON |     224 -> 228    |        0 -> 0     
  64  128  16    6   1    ON |     228 -> 232    |        0 -> 0     
  64  128  16    7   0    ON |     234 -> 240    |        0 -> 0     
  64  128  16    7   1    ON |     236 -> 244    |        0 -> 0     
  64  128  16    8   0    ON |     176 -> 214    |        0 -> 0     
  64  128  16    8   1    ON |     180 -> 214    |        0 -> 0     
  80  128  16    1   0   off |     190 -> 190    |        0 -> 0     
  80  128  16    1   1   off |     178 -> 178    |        0 -> 0     
  80  128  16    2   0   off |     188 -> 188    |        0 -> 0     
  80  128  16    2   1   off |     176 -> 176    |        0 -> 0     
  80  128  16    3   0   off |     176 -> 176    |        0 -> 0     
  80  128  16    3   1   off |     176 -> 176    |        0 -> 0     
  80  128  16    4   0   off |     190 -> 190    |        0 -> 0     
  80  128  16    4   1   off |     176 -> 176    |        0 -> 0     
  80  128  16    5   0   off |     178 -> 178    |        0 -> 0     
  80  128  16    5   1   off |     178 -> 178    |        0 -> 0     
  80  128  16    6   0   off |     178 -> 178    |        0 -> 0     
  80  128  16    6   1   off |     178 -> 178    |        0 -> 0     
  80  128  16    7   0   off |     236 -> 236    |        0 -> 0     
  80  128  16    7   1   off |     234 -> 234    |        0 -> 0     
  80  128  16    8   0   off |     186 -> 186    |        0 -> 0     
  80  128  16    8   1   off |     178 -> 178    |        0 -> 0     
  96  128  16    1   0   off |     238 -> 238    |        0 -> 0     
  96  128  16    1   1   off |     234 -> 234    |        0 -> 0     
  96  128  16    2   0   off |     188 -> 188    |        0 -> 0     
  96  128  16    2   1   off |     188 -> 188    |        0 -> 0     
  96  128  16    3   0   off |     242 -> 242    |        0 -> 0     
  96  128  16    3   1   off |     240 -> 240    |        0 -> 0     
  96  128  16    4   0   off |     182 -> 182    |        0 -> 0     
  96  128  16    4   1   off |     182 -> 182    |        0 -> 0     
  96  128  16    5   0   off |     250 -> 250    |        0 -> 0     
  96  128  16    5   1   off |     246 -> 246    |        0 -> 0     
  96  128  16    6   0   off |     250 -> 250    |        0 -> 0     
  96  128  16    6   1   off |     246 -> 246    |        0 -> 0     
  96  128  16    7   0   off |     248 -> 248    |        0 -> 0     
  96  128  16    7   1   off |     254 -> 254    |        0 -> 0     
  96  128  16    8   0   off |     178 -> 178    |        0 -> 0     
  96  128  16    8   1   off |     178 -> 178    |        0 -> 0     

THE TARGET'S OWN TILE. 5120x1024 pins K32N128, whose ladder is 16x128 (kernel
shape 2, M <= 16) and 32x128 (shape 5, M 17-32). The checkpoint's k/v_proj is
K=5 bits, so those are the four rows that decide this change:

  16x128 TK32 bits5: regs 122 -> 128 / 122 -> 127,
                     spill 0 -> 0 / 0 -> 0   (fp16 / fp32 C)

Other bit widths at the same tile are NOT all spill-free -- read the table, not
this summary. Quoting one bit width as if it were the tile was a mistake made
once already in drafting this receipt.

RESIDUAL. Structural, not an addressing choice: the reducer holds every
accumulator live across a variable-trip-count loop of global loads. That is
irreducible for any fixup that folds L partials into registers. Tiles whose block
is 512 threads are pinned at 128 registers by the 65536-per-SM budget, so ptxas
cannot give them more and takes the difference as spill -- in `reduce()`, which
runs once per output column, not in the mainloop. That is the row to watch when
this is finally timed.


WHERE THE REGISTERS ACTUALLY GO -- a probe, not an inference
===========================================================

The residual was attributed above to "the reducer holds every accumulator live
across a variable-trip-count loop of global loads". That is a mechanism, so it is
falsifiable: replace the loop with exactly ONE fold and the cost should vanish.
Third arm below, `one` -- identical in every other respect, and deliberately
WRONG (it folds one partial where L-1 are due), built only to be compiled.

Bit width 5, C fp16 / fp32, only the instantiations the gate turns ON:

  TM   TN  TK f32 |       chain        loop    one fold   (regs/spill)
  16  128  16   0 |       128/0       158/0       128/0
  16  128  16   1 |       126/0       158/0       128/0
  16  128  32   0 |       122/0       128/0       124/0
  16  128  32   1 |       122/0       127/0       124/0
  16  256  32   0 |      128/48      128/56      128/56
  16  256  32   1 |      128/48      128/56      128/56
  16  512  16   0 |       196/0       206/0       204/0
  16  512  16   1 |       206/0       210/0       210/0
  32  128  16   0 |      128/40       166/0      128/48
  32  128  16   1 |       160/0       166/0       164/0
  32  128  32   0 |      128/40      128/48      128/48
  32  128  32   1 |      128/48      128/80      128/60
  32  256  16   0 |       186/0       198/0       196/0
  32  256  16   1 |       200/0       204/0       206/0
  48  128  16   0 |       176/0       200/0       176/0
  48  128  16   1 |       176/0       208/0       177/0
  64  128  16   0 |       224/0       228/0       228/0
  64  128  16   1 |       228/0       232/0       232/0

CONFIRMED, and more sharply than expected. Folding one partial is within 0-2
registers of the chain on nearly every shape -- 48x128 returns exactly to its
baseline 176/0 from the loop's 200/0, and 16x128 TK16 exactly to 128/0 from
158/0. The staging store itself is nearly free; ALL of the cost is the loop.

WHAT THAT IMPLIES FOR THE DESIGN, and it is not what was built.

A PAIRWISE TREE folds exactly one partial per level. It would take L from 16
serialised round trips to log2(L) = 4 -- about 65% of the one-shot fixup's
available win, using the same arena and the same release/acquire protocol, with
only the lock arithmetic changed -- at the register cost measured in the `one
fold` column, i.e. approximately zero.

So the two designs are a real trade and the numbers to compare are:

    one-shot fixup   L -> 1 wait      +4 to +32 registers, some new spill
    pairwise tree    L -> log2 L      ~0 registers

The one-shot is what is built and what should be timed first, because if its
register cost does not cost time then it is strictly better. The tree is the
fallback, and this probe is the reason it is a cheap one rather than a rewrite.


POSTSCRIPT, after the card run (docs/receipts/exl3-streamk-fixup-bench.txt)
===========================================================================

The register census above is sound and its conclusion is beside the point.

The one-shot fixup was measured against the chain on an RTX 4090 and loses at the
served slates -- -3.10% of leg A at M=48, -4.22% at M=64 -- and the binding cost
is NOT the +4 to +32 registers or the spill this file catalogues. It is staging
TRAFFIC: each contributor writes a full (size_m x TILESIZE_N) partial and the
reducer reads L-1 of them, so the fixup moves 2*L*M*TILESIZE_N*4 bytes per output
column. At M=64, TILESIZE_N=128, L=16 that is 7.7 MB against a 3.28 MB trellis
read, on a kernel that is bandwidth-bound. Half of that is self-inflicted: the
partials are fp32 where the chain stages in C's fp16.

That also retires the pairwise tree named above as the fallback. The tree trades
latency for register pressure, and latency is not what binds at M >= 48 -- it
would move the same bytes as the one-shot and lose for the same reason. The only
variant with a live case is an fp16-partial fixup, which would move exactly the
chain's bytes; it was not built, and it would still have to beat force_num_sms=64
rather than the default chain.

Read this file as what it is: a measurement of one axis that turned out not to be
the deciding one. The register numbers are still the right answer to the question
they were asked.
