int8 VERIFY leg on a small-M tile: the served chain against leg A
=================================================================

SUPERSEDED ON THE MIN-ROWS QUESTION (added 2026-09-10).  Everything below
scores the int8 chain against `legA = exl3_gemm(force_shape_idx=-1)`, an
UNPINNED dispatch no boot serves.  Measured against the leg the engine
actually serves, the crossing is 36 on all seven geometries
(int8-verify-slate-geometry-2026-09-07.txt), which is the value
`exl3_int8_verify_min_rows` ships.  The "64 is the default" and "40" readings
below are both historical.  Read this file for the tile mechanism; read
int8-verify-slate-geometry-2026-09-07.txt and
int8-verify-min-rows-ab-2026-09-07.txt for the threshold.

WHAT THIS DECIDES.  `ARBI_EXL3_INT8_VERIFY=1` routes a B*(K+1) speculative
slate to the int8 EXL3 GEMM.  Until this branch it ran that slate on the
PREFILL pin -- TILE_M=128 -- so every call padded into one m-tile and the
launcher's (N/TILE_N)*ceil(M/TILE_M) grid launched N/TILE_N blocks on a 128-SM
card while still reading the whole trellis.  The branch pins the class to its
own narrow tile.  This is the measurement of whether that is enough to beat
the leg the class takes today.

IT IS NOT, at the row counts a max_batch-8 DFlash2 K=7 deployment actually
captures.  Leg A wins outright at M=8 and M=16 on all three linears, on every
tile.  At M=32 it still wins on out_proj and down_proj and loses only on
gate_proj.  The int8 chain wins on all three only at M=64 -- i.e. only at full
concurrency, which is the regime speculative decoding is least valuable in.
`exl3_int8_verify` therefore does not have one right setting -- it has a
crossover.  It defaults to `auto`, which serves the slates above
`exl3_int8_verify_min_rows` (64 when this was written; the CORRECTION below
moves it to 40) and refuses the rest by name; the tile default is a
measurement rather than a placeholder.  The end-to-end section below is
the same crossover on the served stack.

CORRECTION (2026-09-05, same day): THE LADDER CAPTURES EVERY B.
  Two paragraphs below -- "WHERE THE CROSSOVER IS" and "WHAT THAT MAKES THE
  DEFAULT" -- say that a max_batch-8 K=7 deployment captures B in {1,2,4,8},
  so M is in {8,16,32,64} and 40/48/56 are eager-only. THAT IS WRONG. The
  canonical boot's own log reads `decode graphs: 16 shapes ... (B=1..8, S=1..8)`
  and `cudagraph: decode ladder covers every schedulable shape (B=1..8,
  S=[1, 8], 1 bucket class(es), pad_up=True)`: every B from 1 to 8 has its own
  captured verify graph at S=K+1=8, so the captured slate widths are
  M in {8,16,24,32,40,48,56,64} and nothing pads up to 64.

  WHAT THAT CHANGES. Every number in this file stands -- the sweep measured
  the real geometries at the real widths. What moves is the conclusion drawn
  from them. B=5,6,7 (M=40,48,56) are SERVED rungs, not interpolation, and the
  "WHERE THE CROSSOVER IS" table says the int8 chain WINS all three of them on
  the tile that wins there: 0.84 / 0.80 on s98 at M=40/48, 0.87 on s94 at
  M=56. A single frozen verify tile at min_rows=64 leaves those three rungs on
  leg A while a tile that wins them is compiled and available.

  SO THE VERIFY PIN IS A LADDER OVER SLATE-ROW BANDS, not one tile: s98
  (TILE_M=16) for rows <= 48, s94 (TILE_M=32) above it -- each band taking the
  tile that wins at the widths it covers, off one reduction key. The band edge
  is `INT8_VERIFY_BAND_ROWS`, overridable per configuration as the shipped
  table's entry-wide `int8_verify_bands`. `exl3_int8_verify_min_rows` moves
  with it, 64 -> 40: that is the crossing THIS table measures on the narrow
  band's own tile (s98 crosses 1.00 between M=32 at 1.20 and M=40 at 0.84 and
  stays below it), where 64 was the crossing for a deployment whose only
  served rung above 32 was 64. VERDICT 2 below reads "s98 leads at M<=32,
  where the leg loses anyway" -- that is still true, and it is why the narrow
  band starts at 40 rather than at 8.

CONFIG
  card            RTX 4090 (sm89, 128 SM), sole tenancy, sm clock median
                  2715 MHz under load, power median 356 W, hw_slowdown 0
  model           Qwen3.8-27B-exl3-4.0bpw, mcg codebook, layer 0
  kernel          arbi_serve/weight_quant/exl3/csrc/exl3_i8_gemm.cu as baked
                  into registry.arbi.work/arbi-serve:test-latest (this branch
                  does not touch the .cu, so the binary is the served one)
  harness         tools/int8_gemm/bench_group_scale.py, reps=7, burst>=30 ms,
                  L2 scrub 'read', ratios pooled WITHIN a rep over a
                  round-robin reshuffled each rep
  rot             6 on out_proj, 2 on gate_proj/down_proj -- enough trellis
                  copies to exceed the 72 MB L2, so a single-shape loop cannot
                  serve the whole weight out of cache

THE ARMS, and why the third one had to be added
  legA    exl3_gemm, the trellis leg.  THE INCUMBENT: below
          auto_reconstruct_threshold the dispatch takes it, and a verify slate
          is below it at every served concurrency.
  legB    reconstruct + cuBLAS hgemm.  Never competitive here; carried as the
          leg the int8 work originally replaced.
  a<sid>  THE SERVED CHAIN: int8_gemm_adapter -- one a_prep_frag pass (suh
          pre-scale, 128-wide Hadamard, per-token/per-group amax, int8
          quantise, fragment-order store) then the GEMM with svh folded into
          its epilogue.  Scratch bound once, outside the timed region.
  g<sid>  the GEMM alone, against a scale table somebody else prepared.  Prices
          no A side, so it flatters the leg.
  f<sid>  the HARNESS chain: standalone had_r_128, separate amax, separate
          prep_a, second had_r_128 on the output.  Four passes the served leg
          does not run.  At a prefill chunk the difference is a rounding error
          on a bandwidth-bound GEMM; at a verify slate the GEMM is tens of
          microseconds and the passes are most of the answer, which is why a
          verdict taken off `f` would be a verdict about a chain nothing
          serves.

TILES
  s111  128x128x128 1x4 gs128   the PREFILL pin (255 regs, 2 blk/SM)
  s85    64x128x128 1x8 gs128
  s94    32x128x128 1x8 gs128   <-- the new verify default
  s98    16x128x128 1x8 gs128
  Ungrouped twins s110/s86/s95/s99 are compiled and probed; the served
  quantiser is exl3_int8_group_scale=1, so the grouped row is what serves.

ROW INVARIANCE, ON THE CARD
  The harness refuses to time entries of one activation-group width that are
  not bit-identical.  s111, s98, s94 and s85 passed that witness at every M
  below, on real 27B weights.  That is the on-card half of what
  int8_reduction_key asserts against the table: TILE_M moves the block count
  and not the arithmetic, which is what makes a per-class tile admissible at
  all.

SERVED CHAIN vs LEG A  (served/legA; <1.00 means the int8 leg is faster)

  M     linear      s111     s98      s94      s85    | legA us  legB us
  ----  ---------  -------  -------  -------  ------- | -------  -------
  8     out_proj   3.5782   1.4819   1.7604   2.2465 |   23.60   131.64   best s98 legA
  8     gate_proj  2.0123   1.0182   1.0521   1.2940 |   52.83   489.36   best s98 legA
  8     down_proj  4.1420   1.6497   1.9720   2.5634 |   55.47   486.25   best s98 legA
  16    out_proj   3.4959   1.4598   1.7278   2.1913 |   24.22   131.86   best s98 legA
  16    gate_proj  1.9895   1.0077   1.0412   1.2816 |   53.56   495.85   best s98 legA
  16    down_proj  3.8553   1.5310   1.8395   2.3821 |   59.70   489.50   best s98 legA
  32    out_proj   2.9068   1.2048   1.4803   1.8247 |   29.34   130.34   best s98 legA
  32    gate_proj  1.6392   1.0091   0.8547   1.0624 |   65.90   483.99   best s94 INT8
  32    down_proj  3.3922   1.3528   1.6843   2.0994 |   67.95   480.39   best s98 legA
  64    out_proj   1.6799   1.0299   0.8545   1.0563 |   51.28   119.21   best s94 INT8
  64    gate_proj  0.9018   0.8896   0.7125   0.5791 |  123.29   490.06   best s85 INT8
  64    down_proj  1.8627   1.1438   0.9223   1.1550 |  124.36   486.12   best s94 INT8


WHERE THE CROSSOVER IS, and why it is not one point
  out_proj, sweeping the rows between the two captured rungs:

    M      s98      s94      s85    | legA us
    ----  -------  -------  ------- | -------
    32    1.2048   1.4803   1.8247  |   29.34
    40    0.8394   1.0152   1.2449  |   43.35
    48    0.8036   0.9862   1.2151  |   44.52
    56    1.0500   0.8659   1.0746  |   50.60
    64    1.0299   0.8545   1.0563  |   51.28

  BOTH LEGS ARE SAWTOOTH IN M and the teeth are out of phase.  The int8 tile
  pads M up to a whole number of TILE_M, so s98 is cheapest where M is a
  multiple of 16 and s94 where it is a multiple of 32; leg A walks its own
  tile ladder and steps at its own boundaries (legA jumps 29 -> 43 us between
  M=32 and M=40, then is nearly flat to M=64).  So the winner alternates, and
  no frozen tile leads across the span.  That is a property of the two
  dispatches, not noise: the paired intervals here are a few tenths of a
  percent wide.

  It also means the intermediate rows are not the ones to optimise.  A
  max_batch-8 K=7 deployment captures B in {1,2,4,8}, i.e. M in {8,16,32,64}
  and nothing between, so M=40/48/56 are eager-only.

WHAT THE PER-CLASS PIN ITSELF BOUGHT
  out_proj, served chain, us/call:

    M     s111 (prefill pin)   s94 (verify pin)   ratio
    ----  ------------------   ----------------   -----
    8            84.47              41.55         2.03x
    16           84.66              41.85         2.02x
    32           85.14              43.43         1.96x
    64           86.23              43.81         1.97x

  So the tile pin is worth about 2x on the leg, consistently, and is what
  brings the leg from 1.7-4.1x SLOWER than leg A to within reach of it.  It is
  necessary and it is not sufficient.

WHAT a_prep COSTS, and why `f` overstates it
  out_proj, M=8:  g98 32.99   a98 34.97   f98 61.26 us/call  (a-g = 1.98)
  out_proj, M=64: g94 39.70   a94 43.81   f94 63.60 us/call  (a-g = 4.11)

  The fused a_prep_frag pass costs ~2-4 us; the harness chain's three separate
  passes plus the unfused output Hadamard cost ~20-28.  Reading the verify
  decision off `f` would have charged the leg roughly six times its real A-side
  cost and buried the finding that the GEMM itself is competitive.

END TO END, ON THE SERVED STACK
  The kernel sweep above is per-linear.  This is the same question asked of
  the whole server, which is what the flag default has to answer to.

  CONFIG   Qwen3.8-27B-exl3-4.0bpw + DFlash2 K=7 + tkv k4v4, --max-batch 8,
           one RTX 4090, 16 fixed reasoning prompts, seeded.  Both arms on
           this branch's commit; only ARBI_EXL3_INT8_VERIFY differs.

    conc   verify M   int8 on   int8 off   delta      accept (on / off)
    ----   --------   -------   --------   --------   -----------------
    c=1        8        97.6      113.5    -14.0%      0.379 / 0.357
    c=8       64        71.5       57.3    +24.7%      0.379 / 0.357

  tok/s is PER STREAM.  Acceptance is unchanged across the arms, so the swing
  is the GEMM leg and not the drafter: at c=1 the slate is 8 rows and the int8
  chain loses by about what the M=8 rows above predict, and at c=8 it is 64
  rows and wins by about what the M=64 rows predict.  The kernel sweep and the
  server agree, and they disagree with each other's SIGN across the same
  span -- which is why neither `0` nor `1` is a defensible default.

  WHAT THAT MAKES THE DEFAULT.  `auto`: the int8 leg serves a VERIFY forward
  whose own slate reaches `exl3_int8_verify_min_rows`, and refuses it by name
  ("verify slate below min rows" on exl3_int8_verify_gemm) below that.  64 is
  the default because it is the first rung this config captures at which the
  leg wins on all three linears AND wins end to end.  A max_batch-8 K=7
  deployment captures B in {1,2,4,8}, i.e. M in {8,16,32,64}, so the rule
  separates exactly the B=8 rung from the rest; an intermediate B pads UP to
  the next captured graph, so the decision is the padded rung's.

VERDICT
  1. `exl3_int8_verify` DEFAULTS TO `auto`, not to a fixed leg.  At the
     concurrencies where speculative decoding earns its keep (B=1..4,
     M=8..32) the trellis leg is 1.20-1.65x faster on out_proj and down_proj,
     and the best case anywhere in that range is a tie on gate_proj -- and the
     end-to-end table above puts that at -14% per stream at c=1.  `auto`
     declines those slates.  `0` (never) remains the setting for a deployment
     that wants verify to take exactly decode's arithmetic at every width.
  2. The verify tile default is s94 (32-row, grouped) / s95 (its per-token
     twin).  Not because it leads the span -- nothing does -- but because M=64
     is the only rung where the leg beats leg A on all three linears, and s94
     is the tile that wins there (0.85 / 0.71 / 0.92).  s98 leads at M<=32,
     where the leg loses anyway on two of three linears, so pinning for those
     rows would give up the only rows it can win.
  3. A deployment that serves at saturation (B=8 continuously) is where the
     leg pays: about 1.10x on out_proj and down_proj and 1.40x on gate_proj at
     that rung on the kernel, and +24.7% per stream end to end.  `auto` takes
     that rung and only that rung; `1` (always) is the arm that takes it
     together with the c=1 loss.
  4. STILL OPEN: gate_proj is the geometry where the int8 leg is nearest to
     leg A across the whole range (1.02x at M=8 against 1.48x on out_proj).
     Its N is 17408 -- 136 n-tiles against out_proj's 40 -- so the narrow-N
     geometries are grid-starved on the int8 side in a way the wide ones are
     not.  A per-geometry `int8_verify_shape` in shipped_pins.json is the seam
     that would exploit that, and it is not exercised by this sweep.

REPRODUCING
  docker run --rm --gpus all --cpus 4 --entrypoint bash \
    -e TORCH_EXTENSIONS_DIR=<warm a_prep build dir> \
    -e I8_MODEL=/models/Qwen3.8-27B-exl3-4.0bpw \
    -v <worktree>:/w -v /mnt/k8scache/models:/models:ro \
    -w /w registry.arbi.work/arbi-serve:test-latest -lc \
    'python tools/int8_gemm/bench_group_scale.py --rows 8 16 32 64 \
       --shapes 111 98 94 85 --linears out_proj --rot 6 --reps 7 \
       --min-burst-ms 30'

  The harness JIT-builds the int8 GEMM, which is minutes of nvcc across every
  trellis width and launcher shape.  The image bakes that binary at
  /opt/cache-baked/torch_extensions; seeding int8_kernel._MODULE from it makes
  the run start in seconds.  Both build.ninja files name the INSTALLED source
  path, so do NOT put the worktree root on sys.path -- shadowing arbi_serve
  changes the compile command and re-triggers both builds.
