# Gate 3 (real-generation KL) on the NemotronH Lightning-30B-A3B MoE
# checkpoint, K=6, mul1 (#1861). RTX 4090 (sm_89), sole tenant,
# registry.arbi.work/arbi-serve:test-latest, int8 ext narrowed to
# I8_ONLY_BITS=6 I8_ONLY_CB=2.
#
# THIS RUN WAS IMPOSSIBLE UNTIL TODAY. gate3_kl refuses to attribute any arm
# unless REF2 scores EXACTLY 0 against REF, and on this checkpoint it scored
# 1.028e-02: the routed MoE combine accumulates with atomicAdd in an order set
# by a dynamic ticket queue, so two identical forwards disagreed and every
# arm's KL was run-to-run noise (#1922). Run here with
# ARBI_MOE_DETERMINISTIC_COMBINE=1.
#
#   [i8] null control: REF2 vs REF max KL is exactly 0
#   REF2 vs REF paired TV: 0.0000 pp [0.0000, 0.0000]
#
# That is the whole point of the mode, and it held.
#
# COVERAGE. 2320 int8 calls, ZERO fallbacks, and the geometries served are
# exactly the six the census rules servable:
#   {(2688,256): 240, (2688,3712): 460, (2688,4096): 120,
#    (2688,10368): 460, (3712,2688): 460, (4096,2688): 580}
# 117 dense linears, 51.2% of prefill FLOPs per token.
#
# THIS IS A HARNESS NUMBER, NOT A SERVING NUMBER. #1917 is still open, so no
# build exports the leg and gate3_kl reaches the kernel by overriding
# `reconstruct` + `_hgemm_row_blocks`, not through the adapter's dispatch.
#
# ===================================================================
# THE READING. Every margin is quoted against its control, because a margin
# without one is not usable.
#
# PAIRED TV, prefill positions, cluster bootstrap over prompts. TV IS the
# expected speculative accept-rate loss, so read it in percentage points:
#
#   REF2 vs REF   0.0000 pp [0.0000, 0.0000]   <- the null control
#   STD  vs REF   1.8920 pp [0.8948, 3.2195]   <- fp16 leg swap alone
#   INT8 vs REF   2.1105 pp [1.1608, 3.1396]   <- int8
#
# INT8 COSTS 0.22 pp MORE THAN SWAPPING BETWEEN TWO fp16 LEGS, and the two
# intervals overlap over almost their whole length. At n=20 positions over 10
# prompts that excess is NOT RESOLVED. What IS resolved is that it is small:
# the whole int8 reading is the same size as the fp16 leg swap's.
#
# PREFILL KL says the same thing from the other side -- int8's mean and max
# are BELOW the control's, its median is 1.94x it:
#   STD   mean 3.863e-03  p50 1.210e-03  max 2.928e-02
#   INT8  mean 2.888e-03  p50 2.348e-03  max 1.170e-02
#
# AND THE TAIL IS MOSTLY NOT THE KERNEL'S. Spearman rank correlation between
# the control's KL and int8's, across all 20 positions, is 0.765: the
# positions where int8 is worst are largely the positions where SWAPPING
# LEGS is worst, i.e. properties of the prompt, not of the quantiser.
#
# VERSUS THE 27B (K=4, mul1, dense, receipt_gate3_kl.txt):
#   27B        STD 3.166e-04   INT8 9.963e-04   int8 = 3.15x the floor
#   NemotronH  STD 3.863e-03   INT8 2.888e-03   int8 = 0.75x the floor
# Int8 sits BETTER against its floor at K=6 here than at K=4 there. Read that
# with the caveat it deserves: the floor itself is 12.2x LARGER on this
# checkpoint, so this instrument has LESS absolute resolving power, not more.
# The honest summary is "int8's error is no bigger than the fp16 leg swap's
# on this architecture at this width", not "int8 is more accurate at K=6".
#
# CORRECTION, AFTER A CROSS-CHECK FROM THE 27B PER-GROUP WORK: SERVED TV IS
# THE WRONG STATISTIC AND THE SHARPER ONE IS ALREADY IN THIS FILE.
#
# TV is taken AFTER the softmax, which turns a small perturbation of the
# residual stream into a large distributional one. That agent registered a
# null control -- the grouped kernel fed a table with no grouping in it,
# numerically the per-token quantiser to 1e-5 relative at the GEMM output --
# and it still moved 0.57 pp on TV, a third of the effect being looked for.
# Two arms that are numerically identical sit 3.1 pp apart on their TV matrix.
# So the overlap in my TV intervals above is DETERMINISTIC SENSITIVITY, not
# sampling noise, and more prompts would not have closed it.
#
# The lm_head CUT's "rel rms IN" column is the same end-to-end quantity
# measured BEFORE the softmax -- the body's divergence entering the head, a
# continuous L2 instead of a top-k distance:
#
#   REF2 vs REF   0.0000e+00      <- the null control, still exactly zero
#   STD  vs REF   4.4018e-02      <- fp16 leg swap alone
#   INT8 vs REF   5.8060e-02      <- int8
#
#   int8 / leg-swap = 1.319x   (the same pair on TV reads 1.115x)
#
# SO THE READING IS RESOLVED AFTER ALL, and it is worse than TV suggested:
# int8's end-to-end divergence into the head is 1.32x what swapping between
# two fp16 legs costs. The TV table above is kept because it is what a sampler
# sees, but it must not be read as the numerics measure.
#
# WHAT THIS POINT ESTIMATE STILL LACKS is an interval: this column is a mean
# over 20 positions with no per-position spread emitted. The paired op-level
# oracle on `perf/1861-per-group-scale` (`_op_error_report`, scored per class
# against an fp32 reconstruct upstream of the amplifier, 400 classes rather
# than 20 positions) is the instrument that would give one, and its
# `_CLASS_ERR` keying fix -- keyed by CLASS alone, so it was populated by
# whichever arm ran first and every later arm's column was that arm's -- is
# worth taking on its own.
#
# WHICH LEG THESE ROWS TAKE, because it is not the one the row count suggests.
# `custom_op` sends a call to leg B (reconstruct + hgemm) above
# `_auto_reconstruct_threshold(out_features)` -- 280, or 64 for a narrow
# out_features. For every geometry here:
#   at 2048 rows (prefill)   -> leg B, which is what the int8 leg replaces
#   at 48/64 rows (verify)   -> leg A (exl3_gemm)
# So the FAMILY pins and the GRID WIDTH recorded for this checkpoint govern
# leg A, i.e. decode and verify, and are not prefill pins. The int8 leg and
# those pins act on different row bands and neither reading transfers to the
# other.
#
# THE HEAD DOES NOT AMPLIFY HERE. lm_head amp (out/in relative rms) is 0.963
# for INT8 vs REF -- slightly attenuating. The 27B's p16c6 amplification
# finding does not reproduce on this checkpoint.
#
# WHERE THE WORST POSITIONS LIVE. Nine of the ten worst INT8 positions name
# an o_proj or out_proj as the worst linear -- the 4096x2688 geometry, which
# is 29 linears and 10.9% of prefill FLOPs. Worth a look before any per-class
# exclusion is proposed.
#
# DECODE IS NOT A NUMERICS MEASURE HERE and is reported only for completeness:
# once a token flips, every later position is a different context. INT8's
# decode mean KL of 1.002 is that contamination, not a kernel error.
#
# ===================================================================
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (312544 > 262144). Running this sequence through the model will result in indexing errors
[i8] basket: 10 prompts x 4096 tok, domains ['code', 'longctx', 'math', 'sci', 'wiki']
compile-cache MISS: inductor-fx — torch.compile FX graphs + lowerings (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0]
compile-cache MISS: triton-cubin — triton kernel cubins (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0/triton]
compile-cache MISS: tkv-autotune — tkv decode/prefill autotune tables (~min/shape) (re)compiles this boot [/cache/tkv]
compile-cache MISS: arbi-prefill-cute — arbi-prefill CuTeDSL kernel JIT (re)compiles this boot [/cache/cute-dsl]
compile-cache MISS: xgrammar — xgrammar compiled grammars (re)compiles this boot [/cache/xgrammar]
compile-cache MISS: megacache-fx — torch Mega-Cache bundle (Inductor FX + AOTAutograd backend) (re)compiles this boot [/cache/arbi-serve/compile-cache/588802f088df2611.bin]
embed_quant='auto': the input embedding is a Embedding, not a VocabParallelEmbedding (the gather-dequant lives there) — leaving the embedding bf16.
default-pool residency snapshot returned no live ranges — the unpooled-weight fold cannot see what it would move
embed_quant='auto': the input embedding is a Embedding, not a VocabParallelEmbedding (the gather-dequant lives there) — leaving the embedding bf16.
Boot: building KV-cache profile — still running (30s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
persistent fold: scratch.rope holds 4194304 B but the RoPE tables account for 0 B — not folding. Find the other allocator.
[TKV] Cold-boot kernel autotune starting: this is a real, expected one-time cost, not a hang — timing candidate decode-kernel configs for the shapes this deployment needs. Per-cell progress prints below as each one resolves; subsequent boots on this machine with an unchanged config reuse the cached table and skip this. See README.md's 'First boot on a new machine' section for real measured timings and how to skip this cost entirely with a pre-baked image (TKV_BAKE=1).
[TKV autotune] start: kv_cache shape=(1024, 256, 512) data_ptr=71e000000 in_capture=False (total_pages=1024) H_kv=2 num_sms=128 sw=0 buckets=(1024, 4096, 8192, 16384) batches=(1, 2) (capped at max_batch=1) reusable_cells=0 inherited=0 variants=['splitk'] tile_tokens=[4, 8, 16] min_blocks=[0, 3]
[TKV autotune] bucket=  1024 batch=  1: HELD splitk splits=32 tt=4 mb=0 (0.052ms) margin=0.183% sigma=0.897% z=0.58 n=13 inherited=False
[TKV autotune] bucket=  1024 batch=  2: HELD splitk splits=32 tt=4 mb=0 (0.054ms) margin=11.236% sigma=44.009% z=0.76 n=2 inherited=False
[TKV autotune] bucket=  4096 batch=  1: HELD splitk splits=32 tt=4 mb=3 (0.056ms) margin=0.384% sigma=5.072% z=0.21 n=3 inherited=False
[TKV autotune] bucket=  4096 batch=  2: HELD splitk splits=128 tt=16 mb=0 (0.058ms) margin=1.623% sigma=3.423% z=1.02 n=4 inherited=False
[TKV autotune] bucket=  8192 batch=  1: HELD splitk splits=256 tt=4 mb=0 (0.060ms) margin=1.896% sigma=7.004% z=0.75 n=14 inherited=False
[TKV autotune] bucket=  8192 batch=  2: HELD splitk splits=256 tt=8 mb=0 (0.059ms) margin=1.803% sigma=4.530% z=0.97 n=3 inherited=False
[TKV autotune] bucket= 16384 batch=  1: HELD splitk splits=256 tt=4 mb=0 (0.062ms) margin=0.743% sigma=2.291% z=0.86 n=4 inherited=False
[TKV autotune] bucket= 16384 batch=  2: HELD splitk splits=256 tt=4 mb=3 (0.076ms) margin=4.322% sigma=46.859% z=0.28 n=3 inherited=False
[TKV autotune] complete: 8 (batch, bucket) cells swept in 237.70s batch_invariant=0 tile_width_invariant=0
[TKV autotune-provenance] fp=25e35c798fca99ae regime=swept cells=8 sweep_src=2772549a0ef713db tile_tokens=4,8,16 num_splits=32,128,256 min_blocks_per_sm=0,3 resolved=0 held=8 inherited=0
activation profile: the decode probe's FIRST call cost 5295705 allocator events / 264 MiB / 239038.1 ms against 5871 / 2 MiB / 41.1 ms warm — a first-call kernel search inside the probe, not a step. The reported peak AND device time are the warm ones (2 passes, the last is what is reported); the cold numbers are logged so the difference is visible.
Boot: serving-kernel prewarm — still running (30s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
driver.modules_loaded baseline: SEEDED at 134217728 B (0.125 GiB) → /root/.cache/arbi-serve/budget-cache/82676f8a8d5ae82b.modules.json. This boot ran UNGATED — it booked its own bracketed growth, so an unregistered pool or a raw cudaMalloc would be inside that number rather than on driver.residual. Every later boot at this configuration is held to it. Expected ONCE per configuration; if it repeats, the budget cache is not persisting (point ARBI_SERVE_BUDGET_CACHE_DIR at a durable volume) and the guard is inert.
/opt/venv/lib/python3.12/site-packages/torch/__init__.py:1172: FutureWarning: `torch.distributed.reduce_op` is deprecated, please use `torch.distributed.ReduceOp` instead
  return isinstance(obj, torch.Tensor)
/work/arbi_serve/engine/memory_budget/pool_residency.py:366: UserWarning: Accessing the data pointer of FakeTensor is deprecated and will error in PyTorch 2.5. This is almost definitely a bug in your code and will cause undefined behavior with subsystems like torch.compile. Please wrap calls to tensor.data_ptr() in an opaque custom op; If all else fails, you can guard accesses to tensor.data_ptr() on isinstance(tensor, FakeTensor). (Triggered internally at /pytorch/c10/core/StorageImpl.cpp:34.)
  if not _in_default_pool(obj.data_ptr()):
[i8] trellis->path map: 6005 linears
JIT compile AFTER serving-ready [cpp_ext]: exl3_i8_gemm_k6_cb2 cached .so load — a live request paid this compile's latency. This is a boot-warmup coverage gap: extend warmup to pre-compile this kernel/specialization. Counter jit_compile_serving (must-not-fire) at GET /v1/admin/flag_truth.
[i8] prefix cache: server default resolved=True, per-request cache_enabled=False; any cache hit below is refused
[i8] warmup done
  [0] code           REF       4096tok  wall=  1.64s  finish=length  leg=232F/0S acc=232x16/0x32
  [0] code           REF2      4096tok  wall=  1.69s  finish=length  leg=232F/0S acc=232x16/0x32
  [0] code           STD       4096tok  wall=  1.68s  finish=length  leg=0F/232S acc=232x16/0x32
  [0] code           INT8      4096tok  wall=  1.75s  finish=length  leg=0F/232S acc=232x16/0x32
  [1] code           REF       4096tok  wall=  1.68s  finish=length  leg=232F/0S acc=232x16/0x32
  [1] code           REF2      4096tok  wall=  1.61s  finish=length  leg=232F/0S acc=232x16/0x32
  [1] code           STD       4096tok  wall=  1.69s  finish=length  leg=0F/232S acc=232x16/0x32
  [1] code           INT8      4096tok  wall=  1.74s  finish=length  leg=0F/232S acc=232x16/0x32
  [2] wiki           REF       4096tok  wall=  1.70s  finish=length  leg=232F/0S acc=232x16/0x32
  [2] wiki           REF2      4096tok  wall=  1.47s  finish=length  leg=232F/0S acc=232x16/0x32
  [2] wiki           STD       4096tok  wall=  1.71s  finish=length  leg=0F/232S acc=232x16/0x32
  [2] wiki           INT8      4096tok  wall=  1.66s  finish=length  leg=0F/232S acc=232x16/0x32
  [3] wiki           REF       4096tok  wall=  1.84s  finish=length  leg=232F/0S acc=232x16/0x32
  [3] wiki           REF2      4096tok  wall=  1.61s  finish=length  leg=232F/0S acc=232x16/0x32
  [3] wiki           STD       4096tok  wall=  1.71s  finish=length  leg=0F/232S acc=232x16/0x32
  [3] wiki           INT8      4096tok  wall=  1.68s  finish=length  leg=0F/232S acc=232x16/0x32
  [4] math           REF       4096tok  wall=  1.55s  finish=length  leg=232F/0S acc=232x16/0x32
  [4] math           REF2      4096tok  wall=  1.72s  finish=length  leg=232F/0S acc=232x16/0x32
  [4] math           STD       4096tok  wall=  1.61s  finish=length  leg=0F/232S acc=232x16/0x32
  [4] math           INT8      4096tok  wall=  1.66s  finish=length  leg=0F/232S acc=232x16/0x32
  [5] math           REF       4096tok  wall=  1.57s  finish=length  leg=232F/0S acc=232x16/0x32
  [5] math           REF2      4096tok  wall=  1.59s  finish=length  leg=232F/0S acc=232x16/0x32
  [5] math           STD       4096tok  wall=  1.62s  finish=length  leg=0F/232S acc=232x16/0x32
  [5] math           INT8      4096tok  wall=  1.69s  finish=length  leg=0F/232S acc=232x16/0x32
  [6] longctx        REF       4096tok  wall=  1.70s  finish=length  leg=232F/0S acc=232x16/0x32
  [6] longctx        REF2      4096tok  wall=  1.55s  finish=length  leg=232F/0S acc=232x16/0x32
  [6] longctx        STD       4096tok  wall=  1.61s  finish=length  leg=0F/232S acc=232x16/0x32
  [6] longctx        INT8      4096tok  wall=  1.49s  finish=length  leg=0F/232S acc=232x16/0x32
  [7] longctx        REF       4096tok  wall=  1.45s  finish=length  leg=232F/0S acc=232x16/0x32
  [7] longctx        REF2      4096tok  wall=  1.44s  finish=length  leg=232F/0S acc=232x16/0x32
  [7] longctx        STD       4096tok  wall=  1.44s  finish=length  leg=0F/232S acc=232x16/0x32
  [7] longctx        INT8      4096tok  wall=  1.47s  finish=length  leg=0F/232S acc=232x16/0x32
  [8] sci            REF       4096tok  wall=  1.42s  finish=length  leg=232F/0S acc=232x16/0x32
  [8] sci            REF2      4096tok  wall=  1.40s  finish=length  leg=232F/0S acc=232x16/0x32
  [8] sci            STD       4096tok  wall=  1.41s  finish=length  leg=0F/232S acc=232x16/0x32
  [8] sci            INT8      4096tok  wall=  1.45s  finish=length  leg=0F/232S acc=232x16/0x32
  [9] sci            REF       4096tok  wall=  1.40s  finish=length  leg=232F/0S acc=232x16/0x32
  [9] sci            REF2      4096tok  wall=  1.39s  finish=length  leg=232F/0S acc=232x16/0x32
  [9] sci            STD       4096tok  wall=  1.40s  finish=length  leg=0F/232S acc=232x16/0x32
  [9] sci            INT8      4096tok  wall=  1.44s  finish=length  leg=0F/232S acc=232x16/0x32

[i8] kernel census: int8 calls=2320 rows=4751360 reconstructs SKIPPED=2320 fallback (not K=6) calls=0
[i8] shapes served by the kernel (K,N)->calls: {(2688, 256): 240, (2688, 3712): 460, (2688, 4096): 120, (2688, 10368): 460, (3712, 2688): 460, (4096, 2688): 580}
[i8] excluded projections: none  excluded calls=0

  linear class                                           served   excl  int8 vs fp32 legB vs fp32 int8 vs legB
  backbone.layers.0.mixer.in_proj                            21      0     6.878e-03    1.429e-03    7.028e-03
  backbone.layers.0.mixer.out_proj                           21      0     1.431e-02    1.840e-03    1.443e-02
  backbone.layers.1.mixer.shared_experts.down_proj           21      0     1.188e-02    1.646e-03    1.199e-02
  backbone.layers.1.mixer.shared_experts.up_proj             21      0     3.900e-03    2.263e-03    4.511e-03
  backbone.layers.10.mixer.shared_experts.down_proj          21      0     1.151e-02    1.692e-03    1.164e-02
  backbone.layers.10.mixer.shared_experts.up_proj            21      0     6.116e-03    2.283e-03    6.527e-03
  backbone.layers.11.mixer.in_proj                           21      0     7.209e-03    1.515e-03    7.367e-03
  backbone.layers.11.mixer.out_proj                          21      0     1.400e-02    1.053e-03    1.405e-02
  backbone.layers.12.mixer.k_proj                            21      0     1.435e-02    1.052e-03    1.439e-02
  backbone.layers.12.mixer.o_proj                            21      0     2.576e-02    3.007e-03    2.593e-02
  backbone.layers.12.mixer.q_proj                            21      0     7.992e-03    2.346e-03    8.333e-03
  backbone.layers.12.mixer.v_proj                            21      0     9.821e-03    8.666e-04    9.860e-03
  backbone.layers.13.mixer.shared_experts.down_proj          21      0     1.221e-02    1.712e-03    1.233e-02
  backbone.layers.13.mixer.shared_experts.up_proj            21      0     7.058e-03    2.384e-03    7.448e-03
  backbone.layers.14.mixer.in_proj                           21      0     8.050e-03    1.529e-03    8.194e-03
  backbone.layers.14.mixer.out_proj                          21      0     1.479e-02    1.821e-03    1.490e-02
  backbone.layers.15.mixer.shared_experts.down_proj          21      0     1.208e-02    1.743e-03    1.220e-02
  backbone.layers.15.mixer.shared_experts.up_proj            21      0     7.384e-03    2.458e-03    7.785e-03
  backbone.layers.16.mixer.in_proj                           21      0     8.188e-03    1.545e-03    8.333e-03
  backbone.layers.16.mixer.out_proj                          21      0     1.536e-02    1.881e-03    1.548e-02
  backbone.layers.17.mixer.shared_experts.down_proj          21      0     1.371e-02    1.806e-03    1.383e-02
  backbone.layers.17.mixer.shared_experts.up_proj            21      0     8.708e-03    2.507e-03    9.066e-03
  backbone.layers.18.mixer.in_proj                           21      0     7.851e-03    1.537e-03    8.000e-03
  backbone.layers.18.mixer.out_proj                          21      0     1.437e-02    2.044e-03    1.452e-02
  backbone.layers.19.mixer.k_proj                            21      0     1.150e-02    9.368e-04    1.154e-02
  backbone.layers.19.mixer.o_proj                            21      0     2.976e-02    3.541e-03    2.998e-02
  backbone.layers.19.mixer.q_proj                            21      0     8.111e-03    2.368e-03    8.447e-03
  backbone.layers.19.mixer.v_proj                            21      0     9.223e-03    8.654e-04    9.266e-03
  backbone.layers.2.mixer.in_proj                            21      0     9.892e-03    1.732e-03    1.004e-02
  backbone.layers.2.mixer.out_proj                           21      0     1.464e-02    1.881e-03    1.476e-02
  backbone.layers.20.mixer.shared_experts.down_proj          21      0     1.337e-02    1.770e-03    1.349e-02
  backbone.layers.20.mixer.shared_experts.up_proj            21      0     9.015e-03    2.495e-03    9.355e-03
  backbone.layers.21.mixer.in_proj                           21      0     9.642e-03    1.560e-03    9.768e-03
  backbone.layers.21.mixer.out_proj                          21      0     1.805e-02    1.910e-03    1.815e-02
  backbone.layers.22.mixer.shared_experts.down_proj          21      0     1.364e-02    1.896e-03    1.377e-02
  backbone.layers.22.mixer.shared_experts.up_proj            21      0     9.128e-03    2.507e-03    9.467e-03
  backbone.layers.23.mixer.in_proj                           21      0     8.570e-03    1.534e-03    8.707e-03
  backbone.layers.23.mixer.out_proj                          21      0     1.595e-02    1.925e-03    1.607e-02
  backbone.layers.24.mixer.shared_experts.down_proj          21      0     1.282e-02    1.806e-03    1.295e-02
  backbone.layers.24.mixer.shared_experts.up_proj            21      0     9.204e-03    2.551e-03    9.552e-03
  backbone.layers.25.mixer.in_proj                           21      0     8.408e-03    1.538e-03    8.548e-03
  backbone.layers.25.mixer.out_proj                          21      0     1.453e-02    1.991e-03    1.466e-02
  backbone.layers.26.mixer.k_proj                            21      0     1.253e-02    9.717e-04    1.257e-02
  backbone.layers.26.mixer.o_proj                            21      0     2.580e-02    3.014e-03    2.597e-02
  backbone.layers.26.mixer.q_proj                            21      0     8.856e-03    2.389e-03    9.174e-03
  backbone.layers.26.mixer.v_proj                            21      0     9.859e-03    8.982e-04    9.898e-03
  backbone.layers.27.mixer.shared_experts.down_proj          21      0     1.345e-02    1.845e-03    1.357e-02
  backbone.layers.27.mixer.shared_experts.up_proj            21      0     8.899e-03    2.472e-03    9.236e-03
  backbone.layers.28.mixer.in_proj                           21      0     8.541e-03    1.561e-03    8.683e-03
  backbone.layers.28.mixer.out_proj                          21      0     1.436e-02    2.008e-03    1.450e-02
  backbone.layers.29.mixer.shared_experts.down_proj          21      0     1.281e-02    1.746e-03    1.293e-02
  backbone.layers.29.mixer.shared_experts.up_proj            21      0     7.584e-03    2.419e-03    7.961e-03
  backbone.layers.3.mixer.shared_experts.down_proj           21      0     1.128e-02    1.811e-03    1.142e-02
  backbone.layers.3.mixer.shared_experts.up_proj             21      0     4.574e-03    2.221e-03    5.088e-03
  backbone.layers.30.mixer.in_proj                           21      0     8.128e-03    1.548e-03    8.275e-03
  backbone.layers.30.mixer.out_proj                          21      0     1.420e-02    1.963e-03    1.434e-02
  backbone.layers.31.mixer.shared_experts.down_proj          21      0     1.350e-02    1.732e-03    1.361e-02
  backbone.layers.31.mixer.shared_experts.up_proj            21      0     7.880e-03    2.452e-03    8.256e-03
  backbone.layers.32.mixer.in_proj                           21      0     9.141e-03    1.553e-03    9.274e-03
  backbone.layers.32.mixer.out_proj                          21      0     1.372e-02    1.999e-03    1.386e-02
  backbone.layers.33.mixer.k_proj                            21      0     1.390e-02    9.931e-04    1.393e-02
  backbone.layers.33.mixer.o_proj                            21      0     2.115e-02    2.274e-03    2.127e-02
  backbone.layers.33.mixer.q_proj                            21      0     9.285e-03    2.426e-03    9.598e-03
  backbone.layers.33.mixer.v_proj                            21      0     1.212e-02    1.039e-03    1.217e-02
  backbone.layers.34.mixer.shared_experts.down_proj          21      0     1.223e-02    1.719e-03    1.235e-02
  backbone.layers.34.mixer.shared_experts.up_proj            21      0     7.505e-03    2.437e-03    7.895e-03
  backbone.layers.35.mixer.in_proj                           21      0     8.630e-03    1.545e-03    8.767e-03
  backbone.layers.35.mixer.out_proj                          21      0     1.399e-02    1.834e-03    1.411e-02
  backbone.layers.36.mixer.shared_experts.down_proj          21      0     1.243e-02    1.742e-03    1.256e-02
  backbone.layers.36.mixer.shared_experts.up_proj            21      0     6.772e-03    2.423e-03    7.194e-03
  backbone.layers.37.mixer.in_proj                           21      0     8.467e-03    1.569e-03    8.611e-03
  backbone.layers.37.mixer.out_proj                          21      0     1.442e-02    1.818e-03    1.454e-02
  backbone.layers.38.mixer.shared_experts.down_proj          21      0     1.298e-02    1.850e-03    1.311e-02
  backbone.layers.38.mixer.shared_experts.up_proj            21      0     6.255e-03    2.343e-03    6.678e-03
  backbone.layers.39.mixer.in_proj                           21      0     8.764e-03    1.596e-03    8.908e-03
  backbone.layers.39.mixer.out_proj                          21      0     1.516e-02    1.637e-03    1.524e-02
  backbone.layers.4.mixer.in_proj                            21      0     8.876e-03    1.597e-03    9.018e-03
  backbone.layers.4.mixer.out_proj                           21      0     1.419e-02    1.043e-03    1.423e-02
  backbone.layers.40.mixer.shared_experts.down_proj          21      0     1.166e-02    1.601e-03    1.175e-02
  backbone.layers.40.mixer.shared_experts.up_proj            21      0     5.981e-03    2.311e-03    6.413e-03
  backbone.layers.41.mixer.in_proj                           21      0     9.314e-03    1.504e-03    9.435e-03
  backbone.layers.41.mixer.out_proj                          21      0     1.367e-02    1.780e-03    1.378e-02
  backbone.layers.42.mixer.k_proj                            21      0     1.375e-02    1.023e-03    1.379e-02
  backbone.layers.42.mixer.o_proj                            21      0     2.598e-02    2.089e-03    2.607e-02
  backbone.layers.42.mixer.q_proj                            21      0     9.449e-03    2.357e-03    9.738e-03
  backbone.layers.42.mixer.v_proj                            21      0     1.019e-02    9.304e-04    1.023e-02
  backbone.layers.43.mixer.shared_experts.down_proj          21      0     1.215e-02    1.696e-03    1.227e-02
  backbone.layers.43.mixer.shared_experts.up_proj            21      0     5.497e-03    2.288e-03    5.955e-03
  backbone.layers.44.mixer.in_proj                           21      0     7.783e-03    1.528e-03    7.931e-03
  backbone.layers.44.mixer.out_proj                          21      0     1.382e-02    1.891e-03    1.395e-02
  backbone.layers.45.mixer.shared_experts.down_proj          21      0     1.263e-02    1.763e-03    1.275e-02
  backbone.layers.45.mixer.shared_experts.up_proj            21      0     5.366e-03    2.236e-03    5.813e-03
  backbone.layers.46.mixer.in_proj                           21      0     8.010e-03    1.600e-03    8.167e-03
  backbone.layers.46.mixer.out_proj                          21      0     1.423e-02    1.862e-03    1.435e-02
  backbone.layers.47.mixer.shared_experts.down_proj          21      0     1.317e-02    1.843e-03    1.330e-02
  backbone.layers.47.mixer.shared_experts.up_proj            21      0     5.316e-03    2.203e-03    5.753e-03
  backbone.layers.48.mixer.in_proj                           21      0     7.844e-03    1.600e-03    8.005e-03
  backbone.layers.48.mixer.out_proj                          21      0     1.266e-02    1.684e-03    1.277e-02
  backbone.layers.49.mixer.shared_experts.down_proj          21      0     1.114e-02    1.795e-03    1.129e-02
  backbone.layers.49.mixer.shared_experts.up_proj            21      0     5.048e-03    2.158e-03    5.487e-03
  backbone.layers.5.mixer.k_proj                             21      0     1.219e-02    9.545e-04    1.223e-02
  backbone.layers.5.mixer.o_proj                             21      0     2.730e-02    2.435e-03    2.740e-02
  backbone.layers.5.mixer.q_proj                             21      0     8.491e-03    2.370e-03    8.817e-03
  backbone.layers.5.mixer.v_proj                             21      0     1.097e-02    9.661e-04    1.101e-02
  backbone.layers.50.mixer.in_proj                           21      0     7.741e-03    1.571e-03    7.898e-03
  backbone.layers.50.mixer.out_proj                          21      0     1.020e-02    1.623e-03    1.032e-02
  backbone.layers.51.mixer.shared_experts.down_proj          21      0     8.727e-03    1.381e-03    8.835e-03
  backbone.layers.51.mixer.shared_experts.up_proj            21      0     4.411e-03    2.189e-03    4.924e-03
  backbone.layers.6.mixer.shared_experts.down_proj           21      0     1.261e-02    1.642e-03    1.270e-02
  backbone.layers.6.mixer.shared_experts.up_proj             21      0     5.540e-03    2.274e-03    5.985e-03
  backbone.layers.7.mixer.in_proj                            21      0     9.131e-03    1.558e-03    9.261e-03
  backbone.layers.7.mixer.out_proj                           21      0     1.518e-02    1.871e-03    1.529e-02
  backbone.layers.8.mixer.shared_experts.down_proj           21      0     1.156e-02    1.597e-03    1.167e-02
  backbone.layers.8.mixer.shared_experts.up_proj             21      0     5.748e-03    2.218e-03    6.159e-03
  backbone.layers.9.mixer.in_proj                            21      0     7.768e-03    1.537e-03    7.919e-03
  backbone.layers.9.mixer.out_proj                           21      0     1.509e-02    2.032e-03    1.523e-02
[i8] per-arm census (int8 kernel calls): {'REF': 0, 'REF2': 0, 'STD': 0, 'INT8': 2320}
[i8] STD control: fused-reconstruct override consulted 2320 times

ARM RECEIPTS -- what each arm actually executed, from the engine's
own per-call census.  fused/standalone is the reconstruct variant;
acc16/acc32 is the cuBLAS compute type leg B asked hgemm for.
  arm       legB calls          rows     fused  standalone     acc16     acc32   fp32pin
  REF             2320       4751360      2320           0      2320         0         0
  REF2            2320       4751360      2320           0      2320         0         0
  STD             2320       4751360         0        2320      2320         0         0
  INT8            2320       4751360         0        2320      2320         0         0
  all arms match their expected leg/accumulator signature
  prefill positions aligned: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2] per prompt, identical across all 4 arms
[i8] null control: REF2 vs REF max KL is exactly 0

THE TAIL over PREFILL positions only -- one per prefill chunk, so
every arm scores the SAME context. Decode positions are reported
separately and are NOT a numerics measure: once a token flips, the
arms are reading different contexts. The control proves it -- pooled
over decode, STD (which contains no int8 at all) has the same max KL
as INT8, so a pooled tail cannot attribute anything to the kernel.

arm            n        mean         p50         p95         p99         MAX     argmax flip
REF           20   0.000e+00   0.000e+00   0.000e+00   0.000e+00   0.000e+00 0/20 =   0.00%
REF2          20   0.000e+00   0.000e+00   0.000e+00   0.000e+00   0.000e+00 0/20 =   0.00%
STD           20   3.863e-03   1.210e-03   2.928e-02   2.928e-02   2.928e-02 0/20 =   0.00%
INT8          20   2.888e-03   2.348e-03   1.170e-02   1.170e-02   1.170e-02 1/20 =   5.00%

Tail attribution -- the 10 worst INT8 prefill positions, with the
control's KL at the SAME position. A tail that is positional shows
the control elevated too; a tail that is the kernel's does not.
   prompt  chunk      INT8 KL       STD KL  ratio INT8/STD
        1      0    1.170e-02    2.928e-02            0.40
        6      0    9.086e-03    2.187e-03            4.15
        2      1    5.658e-03    3.968e-03            1.43
        1      1    5.119e-03    7.719e-03            0.66
        6      1    4.543e-03    2.300e-03            1.98
        3      0    3.819e-03    1.210e-03            3.16
        0      0    2.946e-03    1.859e-02            0.16
        7      1    2.873e-03    2.794e-03            1.03
        0      1    2.690e-03    3.106e-04            8.66
        7      0    2.348e-03    2.263e-04           10.37
  Spearman rank correlation STD vs INT8 across all 20 prefill positions: 0.765

Decode positions (divergence-contaminated, shown for completeness):
arm            n        mean         MAX    greedy agree
REF           80   0.000e+00   0.000e+00 80/80 =  100.0%
REF2          80   0.000e+00   0.000e+00 80/80 =  100.0%
STD           80   2.088e-03   1.310e-02 80/80 =  100.0%
INT8          80   1.002e+00   2.333e+01 75/80 =   93.8%

SAMPLER-RELEVANT TAIL @ canonical (T=1.0, top_p=0.95, top_k=20), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF           20     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 20     0.00%
REF2          20     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 20     0.00%
STD           20      97.98%  1.892e-02  1.012e-01  1.012e-01          inf 6/ 20     3.75%
INT8          20      96.71%  2.110e-02  6.823e-02  6.823e-02          inf 9/ 20     7.50%

SAMPLER-RELEVANT TAIL @ owner's k~40 (T=1.0, top_p=1.0, top_k=40), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF           20     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 20     0.00%
REF2          20     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 20     0.00%
STD           20      96.34%  1.874e-02  1.038e-01  1.038e-01          inf 14/ 20     5.62%
INT8          20      95.36%  2.111e-02  6.391e-02  6.391e-02          inf 12/ 20     6.88%

FULL-VOCAB numbers below, kept and labelled: they are the right
instrument for GREEDY or beam decode, where every rank matters.

arm         first-prefill-position KL       median          max
REF                         0.000e+00    0.000e+00    0.000e+00
REF2                        0.000e+00    0.000e+00    0.000e+00
STD                         5.889e-03    1.699e-03    2.928e-02
INT8                        3.270e-03    1.787e-03    1.170e-02

PAIRED TV @ canonical (T=1.0, top_p=0.95, top_k=20), prefill positions.
Each row is A scored against B -- B is that ROW's reference, not the
table's.  TV is the expected speculative accept-rate loss, so the
column is read directly in percentage points.  The interval is a
cluster bootstrap over PROMPTS; the IQR is the raw spread over
positions with no floor applied to it.

  A       vs B        n  mean TV pp           95% CI pp    p25 pp    p50 pp    p75 pp    p95 pp    max pp
  REF2    REF        20      0.0000 [  0.0000,  0.0000]    0.0000    0.0000    0.0000    0.0000    0.0000
  STD     REF        20      1.8920 [  0.8948,  3.2195]    0.0000    1.0308    2.9346   10.1173   10.1173
  INT8    STD        20      2.9179 [  1.9422,  3.8977]    1.2650    2.7957    4.6861    8.2885    8.2885
  INT8    REF        20      2.1105 [  1.1608,  3.1396]    0.0000    1.8433    3.4982    6.8234    6.8234

  TV distance matrix, mean TV pp (symmetric):
          REF      REF2       STD      INT8
  REF        0.0000    0.0000    1.8920    2.1105
  REF2       0.0000    0.0000         -         -
  STD        1.8920         -    0.0000    2.9179
  INT8       2.1105         -    2.9179    0.0000

  TV distance matrix, median TV pp (symmetric):
          REF      REF2       STD      INT8
  REF        0.0000    0.0000    1.0308    1.8433
  REF2       0.0000    0.0000         -         -
  STD        1.0308         -    0.0000    2.7957
  INT8       1.8433         -    2.7957    0.0000

lm_head CUT -- the divergence entering the head vs leaving it,
prefill positions, relative rms.  'in' is the residual stream after
24 layers; 'out' is the centred logits.  amp = out/in: at 1.0 the
head passes the body's divergence through unchanged, above it the
head adds its own.

  A       vs B        n   rel rms IN  rel rms OUT      amp
  REF2    REF        20   0.0000e+00   0.0000e+00      nan
  STD     REF        20   4.4018e-02   4.2681e-02    0.970
  INT8    STD        20   5.8894e-02   5.7240e-02    0.972
  INT8    REF        20   5.8060e-02   5.5901e-02    0.963

PER-TOKEN AMAX -- is the crest factor of the quantised row what
selects the tail?  crest = amax/rms of the int8 GEMM's input row;
under the uniform-residual model the relative noise the row picks
up is crest/(127*sqrt(12)) = crest/440, and qerr is that noise
measured directly by round-tripping the scored row through int8.
arm supplying the statistics: INT8

  statistic                                  p50         p95         p99         max
  crest, scored row (max/linear)       9.183e+00   1.348e+01   1.348e+01   1.348e+01
  crest, any row in chunk              1.786e+01   2.075e+01   2.075e+01   2.075e+01
  qerr, scored row (rel rms)           2.096e-02   3.069e-02   3.069e-02   3.069e-02
  n positions: 20

What the per-token scale is paying for, over the whole census.
crest_g is the crest a per-128-group scale would leave --
sqrt(mean_b amax_b^2)/rms -- and its ratio to crest is the noise
reduction available.  conc is max/median of the per-128-block rms:
at 1 the row is flat and grouping buys nothing.
  statistic                                  p50         p95         p99         max
  crest_g (per-128 scale)                  2.631       2.878       2.961       3.095
  crest / crest_g (noise cut)              1.670       3.268       3.993       4.849
  block rms concentration                  2.087       9.983      19.524    2670.177
  n (class, position) samples: 2320
  heaviest-block STABILITY per class (share of positions whose argmax block is that class's modal block):
    p05=0.150 p50=0.250 p95=0.950 over 116 classes
    a static outlier split needs this near 1.0; at 1/nblocks (~0.025 for K=5120) the heavy block is a different block every token and no static partition exists

The 10 worst INT8 positions, with the crest factor at each.  If the
per-token amax is the mechanism, these are the extreme-crest rows.
   prompt  chunk      INT8 KL   crest row  crest chunk        qerr  worst linear
        1      0    1.170e-02       8.023       15.227   1.819e-02  backbone.layers.33.mixer.o_proj
        6      0    9.086e-03       9.338       17.430   2.100e-02  backbone.layers.5.mixer.o_proj
        2      1    5.658e-03       8.738       19.134   1.989e-02  backbone.layers.42.mixer.o_proj
        1      1    5.119e-03      11.848       16.759   2.682e-02  backbone.layers.42.mixer.o_proj
        6      1    4.543e-03       9.759       18.503   2.219e-02  backbone.layers.5.mixer.o_proj
        3      0    3.819e-03       9.183       18.175   2.096e-02  backbone.layers.7.mixer.out_proj
        0      0    2.946e-03       8.884       17.046   2.006e-02  backbone.layers.50.mixer.out_proj
        7      1    2.873e-03       7.132       18.442   1.618e-02  backbone.layers.42.mixer.o_proj
        0      1    2.690e-03      10.564       16.099   2.377e-02  backbone.layers.42.mixer.o_proj
        7      0    2.348e-03       8.878       17.863   2.032e-02  backbone.layers.42.mixer.o_proj
  ... and the 10 MEDIAN INT8 positions, as the contrast:
        8      0    3.994e-04      10.928       17.435   2.496e-02  backbone.layers.5.mixer.o_proj
        9      0    1.056e-03       8.189       20.748   1.866e-02  backbone.layers.5.mixer.o_proj
        2      0    1.226e-03       8.702       19.178   1.965e-02  backbone.layers.5.mixer.o_proj
        8      1    1.700e-03       9.721       20.025   2.211e-02  backbone.layers.5.mixer.o_proj
        5      1    2.133e-03       6.560       17.647   1.481e-02  backbone.layers.39.mixer.out_proj
        7      0    2.348e-03       8.878       17.863   2.032e-02  backbone.layers.42.mixer.o_proj
        0      1    2.690e-03      10.564       16.099   2.377e-02  backbone.layers.42.mixer.o_proj
        7      1    2.873e-03       7.132       18.442   1.618e-02  backbone.layers.42.mixer.o_proj
        0      0    2.946e-03       8.884       17.046   2.006e-02  backbone.layers.50.mixer.out_proj
        3      0    3.819e-03       9.183       18.175   2.096e-02  backbone.layers.7.mixer.out_proj

Is the tail MORE NOISE or MORE RESPONSE?  rel rms of the residual
stream entering lm_head, per position, and the INT8/STD ratio.
Flat ratio across the tail => the position amplifies every arm and
int8 is merely 2.4x noisier everywhere.  Ratio spiking on the tail
=> int8 injects locally more noise there, which is a mechanism.
  ratio over ALL 20 positions: p50=1.368 p95=2.175 max=2.175
  ratio at the 10 worst INT8 positions: min=0.944 median=1.264 max=2.111
   prompt  chunk      INT8 KL  relIN INT8   relIN STD    ratio
        1      0    1.170e-02  7.9676e-02  7.8455e-02    1.016
        6      0    9.086e-03  7.6526e-02  8.1070e-02    0.944
        2      1    5.658e-03  3.2522e-02  2.3771e-02    1.368
        1      1    5.119e-03  6.2678e-02  5.4602e-02    1.148
        6      1    4.543e-03  3.5496e-02  2.2848e-02    1.554
        3      0    3.819e-03  5.4600e-02  3.2576e-02    1.676
        0      0    2.946e-03  7.7875e-02  7.6006e-02    1.025
        7      1    2.873e-03  3.6277e-02  3.1260e-02    1.160
        0      1    2.690e-03  5.4098e-02  2.5629e-02    2.111
        7      0    2.348e-03  2.9629e-02  1.7507e-02    1.692
  Spearman(relIN ratio, INT8 KL) = -0.209   Spearman(relIN INT8, INT8 KL) = -0.053

Rank correlation of the crest factor with each arm's KL.  STD is the
NULL CONTROL: it runs no quantiser, so the crest factor cannot cause
its KL.  A crest that ranks STD as well as it ranks INT8 is ranking
hard POSITIONS, not the quantiser, and explains nothing.
  crest statistic                    REF       REF2        STD       INT8  INT8 excess
  crest, scored row                0.026      0.026     -0.092     -0.054        0.086
  crest, any row                   0.493      0.493      0.014     -0.054        0.289
  qerr, scored row                 0.060      0.060     -0.144     -0.089        0.108

Mean crest of the scored row, by linear class -- the 15 heaviest.
A class well above the Gaussian-rotation expectation (~4.5 for a
128-wide Hadamard over K=5120) is a class the per-token scale is
wasting levels on.
  linear class                                        mean crest         p95         max
  backbone.layers.5.mixer.o_proj                           7.936      13.479      13.479
  backbone.layers.42.mixer.o_proj                          7.867      11.848      11.848
  backbone.layers.33.mixer.o_proj                          5.808       8.432       8.432
  backbone.layers.19.mixer.o_proj                          5.672       7.061       7.061
  backbone.layers.23.mixer.out_proj                        5.645       8.781       8.781
  backbone.layers.37.mixer.out_proj                        5.605       9.246       9.246
  backbone.layers.21.mixer.out_proj                        5.558       7.041       7.041
  backbone.layers.39.mixer.out_proj                        5.273       8.356       8.356
  backbone.layers.25.mixer.out_proj                        5.232       6.692       6.692
  backbone.layers.12.mixer.o_proj                          5.216       7.718       7.718
  backbone.layers.30.mixer.out_proj                        5.207       7.521       7.521
  backbone.layers.7.mixer.out_proj                         5.150       9.183       9.183
  backbone.layers.44.mixer.out_proj                        5.133       7.023       7.023
  backbone.layers.26.mixer.o_proj                          5.090       6.035       6.035
  backbone.layers.5.mixer.k_proj                           5.062       5.368       5.368
