[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (317092 > 262144). Running this sequence through the model will result in indexing errors
[i8] basket: 6 prompts x 16384 tok, domains ['code', 'longctx', 'math', 'sci', 'wiki']
compile-cache MISS: inductor-fx — torch.compile FX graphs + lowerings (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0]
compile-cache MISS: triton-cubin — triton kernel cubins (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0/triton]
compile-cache MISS: tkv-autotune — tkv decode/prefill autotune tables (~min/shape) (re)compiles this boot [/cache/tkv]
compile-cache MISS: arbi-prefill-cute — arbi-prefill CuTeDSL kernel JIT (re)compiles this boot [/cache/cute-dsl]
compile-cache MISS: xgrammar — xgrammar compiled grammars (re)compiles this boot [/cache/xgrammar]
compile-cache MISS: megacache-fx — torch Mega-Cache bundle (Inductor FX + AOTAutograd backend) (re)compiles this boot [/cache/arbi-serve/compile-cache/588802f088df2611.bin]
Boot: loading model weights — still running (30s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
default-pool residency snapshot returned no live ranges — the unpooled-weight fold cannot see what it would move
exl3 kernel-shape pin: SEEDED 8 geometr(ies) by TIMING them in 518 ms (17408x5120xK4=K32N128, 5120x10240xK4=K32N128, 5120x1024xK4=K32N128, 5120x12288xK4=K32N256, 5120x17408xK4=K16N512, 5120x248320xK6=K16N512, 5120x6144xK4=K32N128, 6144x5120xK4=K32N128). This boot's kernel choice is a fresh measurement, so it need not match the previous boot's. Expected ONCE per configuration; if it repeats, the pin is not being recorded (point ARBI_SERVE_EXL3_PIN_CACHE_DIR at a durable volume) and every cross-boot comparison keeps paying for it.
Boot: building KV-cache profile — still running (30s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
[TKV] Cold-boot kernel autotune starting: this is a real, expected one-time cost, not a hang — timing candidate decode-kernel configs for the shapes this deployment needs. Per-cell progress prints below as each one resolves; subsequent boots on this machine with an unchanged config reuse the cached table and skip this. See README.md's 'First boot on a new machine' section for real measured timings and how to skip this cost entirely with a pre-baked image (TKV_BAKE=1).
[TKV autotune] start: kv_cache shape=(1024, 256, 2048) data_ptr=6a0000000 in_capture=False (total_pages=1024) H_kv=4 num_sms=128 sw=0 buckets=(1024, 4096, 8192, 16384) batches=(1, 2) (capped at max_batch=1) reusable_cells=0 inherited=0 variants=['splitk'] tile_tokens=[4, 8, 16] min_blocks=[0, 3]
[TKV autotune] bucket=  1024 batch=  1: HELD splitk splits=32 tt=4 mb=0 (0.050ms) margin=0.041% sigma=4.930% z=0.02 n=28 inherited=False
[TKV autotune] bucket=  1024 batch=  2: HELD splitk splits=128 tt=8 mb=3 (0.050ms) margin=0.448% sigma=0.689% z=1.47 n=4 inherited=False
[TKV autotune] bucket=  4096 batch=  1: HELD splitk splits=128 tt=8 mb=0 (0.050ms) margin=0.252% sigma=0.708% z=1.03 n=7 inherited=False
[TKV autotune] bucket=  4096 batch=  2: HELD splitk splits=64 tt=4 mb=0 (0.050ms) margin=0.055% sigma=0.416% z=0.33 n=4 inherited=False
[TKV autotune] bucket=  8192 batch=  1: HELD splitk splits=128 tt=4 mb=0 (0.050ms) margin=0.025% sigma=0.566% z=0.10 n=14 inherited=False
[TKV autotune] bucket=  8192 batch=  2: HELD splitk splits=32 tt=8 mb=0 (0.050ms) margin=0.955% sigma=0.492% z=4.95 n=2 inherited=False
[TKV autotune] bucket= 16384 batch=  1: HELD splitk splits=256 tt=8 mb=0 (0.049ms) margin=0.453% sigma=0.892% z=1.33 n=6 inherited=False
[TKV autotune] bucket= 16384 batch=  2: HELD splitk splits=32 tt=4 mb=0 (0.152ms) margin=0.029% sigma=0.080% z=1.06 n=2 inherited=False
[TKV autotune] complete: 8 (batch, bucket) cells swept in 78.82s batch_invariant=0 tile_width_invariant=0
[TKV autotune-provenance] fp=57911e26f4e47f74 regime=swept cells=8 sweep_src=2772549a0ef713db tile_tokens=4,8 num_splits=32,64,128,256 min_blocks_per_sm=0,3 resolved=0 held=8 inherited=0
activation profile: the decode probe's FIRST call cost 8334532 allocator events / 1036 MiB / 80659.9 ms against 13305 / 1 MiB / 48.1 ms warm — a first-call kernel search inside the probe, not a step. The reported peak AND device time are the warm ones (2 passes, the last is what is reported); the cold numbers are logged so the difference is visible.
driver.modules_loaded baseline: SEEDED at 144703488 B (0.135 GiB) → /root/.cache/arbi-serve/budget-cache/69b5561ff6f9c4cf.modules.json. This boot ran UNGATED — it booked its own bracketed growth, so an unregistered pool or a raw cudaMalloc would be inside that number rather than on driver.residual. Every later boot at this configuration is held to it. Expected ONCE per configuration; if it repeats, the budget cache is not persisting (point ARBI_SERVE_BUDGET_CACHE_DIR at a durable volume) and the guard is inert.
/opt/venv/lib/python3.12/site-packages/torch/__init__.py:1172: FutureWarning: `torch.distributed.reduce_op` is deprecated, please use `torch.distributed.ReduceOp` instead
  return isinstance(obj, torch.Tensor)
/work/arbi_serve/engine/memory_budget/pool_residency.py:366: UserWarning: Accessing the data pointer of FakeTensor is deprecated and will error in PyTorch 2.5. This is almost definitely a bug in your code and will cause undefined behavior with subsystems like torch.compile. Please wrap calls to tensor.data_ptr() in an opaque custom op; If all else fails, you can guard accesses to tensor.data_ptr() on isinstance(tensor, FakeTensor). (Triggered internally at /pytorch/c10/core/StorageImpl.cpp:34.)
  if not _in_default_pool(obj.data_ptr()):
[i8] trellis->path map: 401 linears
JIT compile AFTER serving-ready [cpp_ext]: exl3_i8_gemm fresh ninja/nvcc build — a live request paid this compile's latency. This is a boot-warmup coverage gap: extend warmup to pre-compile this kernel/specialization. Counter jit_compile_serving (must-not-fire) at GET /v1/admin/flag_truth.
[i8] prefix cache: server default resolved=True, per-request cache_enabled=False; any cache hit below is refused
JIT compile AFTER serving-ready [cpp_ext]: arbi_serve_exl3_a_prep_v3 cached .so load — a live request paid this compile's latency. This is a boot-warmup coverage gap: extend warmup to pre-compile this kernel/specialization. Counter jit_compile_serving (must-not-fire) at GET /v1/admin/flag_truth.
[i8] warmup done
  [0] code           REF     16384tok  wall=  4.91s  finish=length
  [0] code           INT8F   16384tok  wall=  3.11s  finish=stop
  [0] code           INT8FO  16384tok  wall=  3.11s  finish=stop
  [1] code           REF     16384tok  wall=  4.69s  finish=stop
  [1] code           INT8F   16384tok  wall=  3.18s  finish=stop
  [1] code           INT8FO  16384tok  wall=  3.18s  finish=stop
  [2] wiki           REF     16384tok  wall=  4.89s  finish=length
  [2] wiki           INT8F   16384tok  wall=  3.40s  finish=length
  [2] wiki           INT8FO  16384tok  wall=  3.39s  finish=length
  [3] math           REF     16384tok  wall=  4.90s  finish=length
  [3] math           INT8F   16384tok  wall=  3.41s  finish=length
  [3] math           INT8FO  16384tok  wall=  3.42s  finish=length
  [4] longctx        REF     16384tok  wall=  4.93s  finish=length
  [4] longctx        INT8F   16384tok  wall=  3.41s  finish=length
  [4] longctx        INT8FO  16384tok  wall=  3.42s  finish=length
  [5] sci            REF     16384tok  wall=  4.94s  finish=length
  [5] sci            INT8F   16384tok  wall=  3.42s  finish=length
  [5] sci            INT8FO  16384tok  wall=  3.43s  finish=length

[i8] kernel census: int8 calls=38400 rows=78643200 reconstructs SKIPPED=38400 fallback (non-4bpw) calls=0
[i8] shapes served by the kernel (K,N)->calls: {(5120, 1024): 3072, (5120, 6144): 4608, (5120, 10240): 4608, (5120, 12288): 1536, (5120, 17408): 12288, (6144, 5120): 6144, (17408, 5120): 6144}
[i8] excluded projections: none  excluded calls=0

  linear class                                           served   excl  int8 vs fp32 legB vs fp32 int8 vs legB
  model.layers.0.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.0.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.0.linear_attn.out_proj                        98      0             -            -            -
  model.layers.0.mlp.down_proj                               98      0             -            -            -
  model.layers.0.mlp.gate_proj                               98      0             -            -            -
  model.layers.0.mlp.up_proj                                 98      0             -            -            -
  model.layers.1.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.1.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.1.linear_attn.out_proj                        98      0             -            -            -
  model.layers.1.mlp.down_proj                               98      0             -            -            -
  model.layers.1.mlp.gate_proj                               98      0             -            -            -
  model.layers.1.mlp.up_proj                                 98      0             -            -            -
  model.layers.10.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.10.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.10.linear_attn.out_proj                       98      0             -            -            -
  model.layers.10.mlp.down_proj                              98      0             -            -            -
  model.layers.10.mlp.gate_proj                              98      0             -            -            -
  model.layers.10.mlp.up_proj                                98      0             -            -            -
  model.layers.11.mlp.down_proj                              98      0             -            -            -
  model.layers.11.mlp.gate_proj                              98      0             -            -            -
  model.layers.11.mlp.up_proj                                98      0             -            -            -
  model.layers.11.self_attn.k_proj                           98      0             -            -            -
  model.layers.11.self_attn.o_proj                           98      0             -            -            -
  model.layers.11.self_attn.q_proj                           98      0             -            -            -
  model.layers.11.self_attn.v_proj                           98      0             -            -            -
  model.layers.12.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.12.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.12.linear_attn.out_proj                       98      0             -            -            -
  model.layers.12.mlp.down_proj                              98      0             -            -            -
  model.layers.12.mlp.gate_proj                              98      0             -            -            -
  model.layers.12.mlp.up_proj                                98      0             -            -            -
  model.layers.13.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.13.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.13.linear_attn.out_proj                       98      0             -            -            -
  model.layers.13.mlp.down_proj                              98      0             -            -            -
  model.layers.13.mlp.gate_proj                              98      0             -            -            -
  model.layers.13.mlp.up_proj                                98      0             -            -            -
  model.layers.14.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.14.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.14.linear_attn.out_proj                       98      0             -            -            -
  model.layers.14.mlp.down_proj                              98      0             -            -            -
  model.layers.14.mlp.gate_proj                              98      0             -            -            -
  model.layers.14.mlp.up_proj                                98      0             -            -            -
  model.layers.15.mlp.down_proj                              98      0             -            -            -
  model.layers.15.mlp.gate_proj                              98      0             -            -            -
  model.layers.15.mlp.up_proj                                98      0             -            -            -
  model.layers.15.self_attn.k_proj                           98      0             -            -            -
  model.layers.15.self_attn.o_proj                           98      0             -            -            -
  model.layers.15.self_attn.q_proj                           98      0             -            -            -
  model.layers.15.self_attn.v_proj                           98      0             -            -            -
  model.layers.16.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.16.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.16.linear_attn.out_proj                       98      0             -            -            -
  model.layers.16.mlp.down_proj                              98      0             -            -            -
  model.layers.16.mlp.gate_proj                              98      0             -            -            -
  model.layers.16.mlp.up_proj                                98      0             -            -            -
  model.layers.17.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.17.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.17.linear_attn.out_proj                       98      0             -            -            -
  model.layers.17.mlp.down_proj                              98      0             -            -            -
  model.layers.17.mlp.gate_proj                              98      0             -            -            -
  model.layers.17.mlp.up_proj                                98      0             -            -            -
  model.layers.18.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.18.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.18.linear_attn.out_proj                       98      0             -            -            -
  model.layers.18.mlp.down_proj                              98      0             -            -            -
  model.layers.18.mlp.gate_proj                              98      0             -            -            -
  model.layers.18.mlp.up_proj                                98      0             -            -            -
  model.layers.19.mlp.down_proj                              98      0             -            -            -
  model.layers.19.mlp.gate_proj                              98      0             -            -            -
  model.layers.19.mlp.up_proj                                98      0             -            -            -
  model.layers.19.self_attn.k_proj                           98      0             -            -            -
  model.layers.19.self_attn.o_proj                           98      0             -            -            -
  model.layers.19.self_attn.q_proj                           98      0             -            -            -
  model.layers.19.self_attn.v_proj                           98      0             -            -            -
  model.layers.2.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.2.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.2.linear_attn.out_proj                        98      0             -            -            -
  model.layers.2.mlp.down_proj                               98      0             -            -            -
  model.layers.2.mlp.gate_proj                               98      0             -            -            -
  model.layers.2.mlp.up_proj                                 98      0             -            -            -
  model.layers.20.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.20.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.20.linear_attn.out_proj                       98      0             -            -            -
  model.layers.20.mlp.down_proj                              98      0             -            -            -
  model.layers.20.mlp.gate_proj                              98      0             -            -            -
  model.layers.20.mlp.up_proj                                98      0             -            -            -
  model.layers.21.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.21.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.21.linear_attn.out_proj                       98      0             -            -            -
  model.layers.21.mlp.down_proj                              98      0             -            -            -
  model.layers.21.mlp.gate_proj                              98      0             -            -            -
  model.layers.21.mlp.up_proj                                98      0             -            -            -
  model.layers.22.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.22.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.22.linear_attn.out_proj                       98      0             -            -            -
  model.layers.22.mlp.down_proj                              98      0             -            -            -
  model.layers.22.mlp.gate_proj                              98      0             -            -            -
  model.layers.22.mlp.up_proj                                98      0             -            -            -
  model.layers.23.mlp.down_proj                              98      0             -            -            -
  model.layers.23.mlp.gate_proj                              98      0             -            -            -
  model.layers.23.mlp.up_proj                                98      0             -            -            -
  model.layers.23.self_attn.k_proj                           98      0             -            -            -
  model.layers.23.self_attn.o_proj                           98      0             -            -            -
  model.layers.23.self_attn.q_proj                           98      0             -            -            -
  model.layers.23.self_attn.v_proj                           98      0             -            -            -
  model.layers.24.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.24.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.24.linear_attn.out_proj                       98      0             -            -            -
  model.layers.24.mlp.down_proj                              98      0             -            -            -
  model.layers.24.mlp.gate_proj                              98      0             -            -            -
  model.layers.24.mlp.up_proj                                98      0             -            -            -
  model.layers.25.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.25.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.25.linear_attn.out_proj                       98      0             -            -            -
  model.layers.25.mlp.down_proj                              98      0             -            -            -
  model.layers.25.mlp.gate_proj                              98      0             -            -            -
  model.layers.25.mlp.up_proj                                98      0             -            -            -
  model.layers.26.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.26.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.26.linear_attn.out_proj                       98      0             -            -            -
  model.layers.26.mlp.down_proj                              98      0             -            -            -
  model.layers.26.mlp.gate_proj                              98      0             -            -            -
  model.layers.26.mlp.up_proj                                98      0             -            -            -
  model.layers.27.mlp.down_proj                              98      0             -            -            -
  model.layers.27.mlp.gate_proj                              98      0             -            -            -
  model.layers.27.mlp.up_proj                                98      0             -            -            -
  model.layers.27.self_attn.k_proj                           98      0             -            -            -
  model.layers.27.self_attn.o_proj                           98      0             -            -            -
  model.layers.27.self_attn.q_proj                           98      0             -            -            -
  model.layers.27.self_attn.v_proj                           98      0             -            -            -
  model.layers.28.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.28.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.28.linear_attn.out_proj                       98      0             -            -            -
  model.layers.28.mlp.down_proj                              98      0             -            -            -
  model.layers.28.mlp.gate_proj                              98      0             -            -            -
  model.layers.28.mlp.up_proj                                98      0             -            -            -
  model.layers.29.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.29.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.29.linear_attn.out_proj                       98      0             -            -            -
  model.layers.29.mlp.down_proj                              98      0             -            -            -
  model.layers.29.mlp.gate_proj                              98      0             -            -            -
  model.layers.29.mlp.up_proj                                98      0             -            -            -
  model.layers.3.mlp.down_proj                               98      0             -            -            -
  model.layers.3.mlp.gate_proj                               98      0             -            -            -
  model.layers.3.mlp.up_proj                                 98      0             -            -            -
  model.layers.3.self_attn.k_proj                            98      0             -            -            -
  model.layers.3.self_attn.o_proj                            98      0             -            -            -
  model.layers.3.self_attn.q_proj                            98      0             -            -            -
  model.layers.3.self_attn.v_proj                            98      0             -            -            -
  model.layers.30.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.30.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.30.linear_attn.out_proj                       98      0             -            -            -
  model.layers.30.mlp.down_proj                              98      0             -            -            -
  model.layers.30.mlp.gate_proj                              98      0             -            -            -
  model.layers.30.mlp.up_proj                                98      0             -            -            -
  model.layers.31.mlp.down_proj                              98      0             -            -            -
  model.layers.31.mlp.gate_proj                              98      0             -            -            -
  model.layers.31.mlp.up_proj                                98      0             -            -            -
  model.layers.31.self_attn.k_proj                           98      0             -            -            -
  model.layers.31.self_attn.o_proj                           98      0             -            -            -
  model.layers.31.self_attn.q_proj                           98      0             -            -            -
  model.layers.31.self_attn.v_proj                           98      0             -            -            -
  model.layers.32.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.32.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.32.linear_attn.out_proj                       98      0             -            -            -
  model.layers.32.mlp.down_proj                              98      0             -            -            -
  model.layers.32.mlp.gate_proj                              98      0             -            -            -
  model.layers.32.mlp.up_proj                                98      0             -            -            -
  model.layers.33.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.33.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.33.linear_attn.out_proj                       98      0             -            -            -
  model.layers.33.mlp.down_proj                              98      0             -            -            -
  model.layers.33.mlp.gate_proj                              98      0             -            -            -
  model.layers.33.mlp.up_proj                                98      0             -            -            -
  model.layers.34.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.34.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.34.linear_attn.out_proj                       98      0             -            -            -
  model.layers.34.mlp.down_proj                              98      0             -            -            -
  model.layers.34.mlp.gate_proj                              98      0             -            -            -
  model.layers.34.mlp.up_proj                                98      0             -            -            -
  model.layers.35.mlp.down_proj                              98      0             -            -            -
  model.layers.35.mlp.gate_proj                              98      0             -            -            -
  model.layers.35.mlp.up_proj                                98      0             -            -            -
  model.layers.35.self_attn.k_proj                           98      0             -            -            -
  model.layers.35.self_attn.o_proj                           98      0             -            -            -
  model.layers.35.self_attn.q_proj                           98      0             -            -            -
  model.layers.35.self_attn.v_proj                           98      0             -            -            -
  model.layers.36.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.36.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.36.linear_attn.out_proj                       98      0             -            -            -
  model.layers.36.mlp.down_proj                              98      0             -            -            -
  model.layers.36.mlp.gate_proj                              98      0             -            -            -
  model.layers.36.mlp.up_proj                                98      0             -            -            -
  model.layers.37.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.37.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.37.linear_attn.out_proj                       98      0             -            -            -
  model.layers.37.mlp.down_proj                              98      0             -            -            -
  model.layers.37.mlp.gate_proj                              98      0             -            -            -
  model.layers.37.mlp.up_proj                                98      0             -            -            -
  model.layers.38.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.38.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.38.linear_attn.out_proj                       98      0             -            -            -
  model.layers.38.mlp.down_proj                              98      0             -            -            -
  model.layers.38.mlp.gate_proj                              98      0             -            -            -
  model.layers.38.mlp.up_proj                                98      0             -            -            -
  model.layers.39.mlp.down_proj                              98      0             -            -            -
  model.layers.39.mlp.gate_proj                              98      0             -            -            -
  model.layers.39.mlp.up_proj                                98      0             -            -            -
  model.layers.39.self_attn.k_proj                           98      0             -            -            -
  model.layers.39.self_attn.o_proj                           98      0             -            -            -
  model.layers.39.self_attn.q_proj                           98      0             -            -            -
  model.layers.39.self_attn.v_proj                           98      0             -            -            -
  model.layers.4.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.4.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.4.linear_attn.out_proj                        98      0             -            -            -
  model.layers.4.mlp.down_proj                               98      0             -            -            -
  model.layers.4.mlp.gate_proj                               98      0             -            -            -
  model.layers.4.mlp.up_proj                                 98      0             -            -            -
  model.layers.40.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.40.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.40.linear_attn.out_proj                       98      0             -            -            -
  model.layers.40.mlp.down_proj                              98      0             -            -            -
  model.layers.40.mlp.gate_proj                              98      0             -            -            -
  model.layers.40.mlp.up_proj                                98      0             -            -            -
  model.layers.41.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.41.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.41.linear_attn.out_proj                       98      0             -            -            -
  model.layers.41.mlp.down_proj                              98      0             -            -            -
  model.layers.41.mlp.gate_proj                              98      0             -            -            -
  model.layers.41.mlp.up_proj                                98      0             -            -            -
  model.layers.42.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.42.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.42.linear_attn.out_proj                       98      0             -            -            -
  model.layers.42.mlp.down_proj                              98      0             -            -            -
  model.layers.42.mlp.gate_proj                              98      0             -            -            -
  model.layers.42.mlp.up_proj                                98      0             -            -            -
  model.layers.43.mlp.down_proj                              98      0             -            -            -
  model.layers.43.mlp.gate_proj                              98      0             -            -            -
  model.layers.43.mlp.up_proj                                98      0             -            -            -
  model.layers.43.self_attn.k_proj                           98      0             -            -            -
  model.layers.43.self_attn.o_proj                           98      0             -            -            -
  model.layers.43.self_attn.q_proj                           98      0             -            -            -
  model.layers.43.self_attn.v_proj                           98      0             -            -            -
  model.layers.44.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.44.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.44.linear_attn.out_proj                       98      0             -            -            -
  model.layers.44.mlp.down_proj                              98      0             -            -            -
  model.layers.44.mlp.gate_proj                              98      0             -            -            -
  model.layers.44.mlp.up_proj                                98      0             -            -            -
  model.layers.45.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.45.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.45.linear_attn.out_proj                       98      0             -            -            -
  model.layers.45.mlp.down_proj                              98      0             -            -            -
  model.layers.45.mlp.gate_proj                              98      0             -            -            -
  model.layers.45.mlp.up_proj                                98      0             -            -            -
  model.layers.46.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.46.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.46.linear_attn.out_proj                       98      0             -            -            -
  model.layers.46.mlp.down_proj                              98      0             -            -            -
  model.layers.46.mlp.gate_proj                              98      0             -            -            -
  model.layers.46.mlp.up_proj                                98      0             -            -            -
  model.layers.47.mlp.down_proj                              98      0             -            -            -
  model.layers.47.mlp.gate_proj                              98      0             -            -            -
  model.layers.47.mlp.up_proj                                98      0             -            -            -
  model.layers.47.self_attn.k_proj                           98      0             -            -            -
  model.layers.47.self_attn.o_proj                           98      0             -            -            -
  model.layers.47.self_attn.q_proj                           98      0             -            -            -
  model.layers.47.self_attn.v_proj                           98      0             -            -            -
  model.layers.48.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.48.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.48.linear_attn.out_proj                       98      0             -            -            -
  model.layers.48.mlp.down_proj                              98      0             -            -            -
  model.layers.48.mlp.gate_proj                              98      0             -            -            -
  model.layers.48.mlp.up_proj                                98      0             -            -            -
  model.layers.49.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.49.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.49.linear_attn.out_proj                       98      0             -            -            -
  model.layers.49.mlp.down_proj                              98      0             -            -            -
  model.layers.49.mlp.gate_proj                              98      0             -            -            -
  model.layers.49.mlp.up_proj                                98      0             -            -            -
  model.layers.5.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.5.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.5.linear_attn.out_proj                        98      0             -            -            -
  model.layers.5.mlp.down_proj                               98      0             -            -            -
  model.layers.5.mlp.gate_proj                               98      0             -            -            -
  model.layers.5.mlp.up_proj                                 98      0             -            -            -
  model.layers.50.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.50.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.50.linear_attn.out_proj                       98      0             -            -            -
  model.layers.50.mlp.down_proj                              98      0             -            -            -
  model.layers.50.mlp.gate_proj                              98      0             -            -            -
  model.layers.50.mlp.up_proj                                98      0             -            -            -
  model.layers.51.mlp.down_proj                              98      0             -            -            -
  model.layers.51.mlp.gate_proj                              98      0             -            -            -
  model.layers.51.mlp.up_proj                                98      0             -            -            -
  model.layers.51.self_attn.k_proj                           98      0             -            -            -
  model.layers.51.self_attn.o_proj                           98      0             -            -            -
  model.layers.51.self_attn.q_proj                           98      0             -            -            -
  model.layers.51.self_attn.v_proj                           98      0             -            -            -
  model.layers.52.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.52.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.52.linear_attn.out_proj                       98      0             -            -            -
  model.layers.52.mlp.down_proj                              98      0             -            -            -
  model.layers.52.mlp.gate_proj                              98      0             -            -            -
  model.layers.52.mlp.up_proj                                98      0             -            -            -
  model.layers.53.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.53.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.53.linear_attn.out_proj                       98      0             -            -            -
  model.layers.53.mlp.down_proj                              98      0             -            -            -
  model.layers.53.mlp.gate_proj                              98      0             -            -            -
  model.layers.53.mlp.up_proj                                98      0             -            -            -
  model.layers.54.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.54.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.54.linear_attn.out_proj                       98      0             -            -            -
  model.layers.54.mlp.down_proj                              98      0             -            -            -
  model.layers.54.mlp.gate_proj                              98      0             -            -            -
  model.layers.54.mlp.up_proj                                98      0             -            -            -
  model.layers.55.mlp.down_proj                              98      0             -            -            -
  model.layers.55.mlp.gate_proj                              98      0             -            -            -
  model.layers.55.mlp.up_proj                                98      0             -            -            -
  model.layers.55.self_attn.k_proj                           98      0             -            -            -
  model.layers.55.self_attn.o_proj                           98      0             -            -            -
  model.layers.55.self_attn.q_proj                           98      0             -            -            -
  model.layers.55.self_attn.v_proj                           98      0             -            -            -
  model.layers.56.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.56.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.56.linear_attn.out_proj                       98      0             -            -            -
  model.layers.56.mlp.down_proj                              98      0             -            -            -
  model.layers.56.mlp.gate_proj                              98      0             -            -            -
  model.layers.56.mlp.up_proj                                98      0             -            -            -
  model.layers.57.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.57.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.57.linear_attn.out_proj                       98      0             -            -            -
  model.layers.57.mlp.down_proj                              98      0             -            -            -
  model.layers.57.mlp.gate_proj                              98      0             -            -            -
  model.layers.57.mlp.up_proj                                98      0             -            -            -
  model.layers.58.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.58.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.58.linear_attn.out_proj                       98      0             -            -            -
  model.layers.58.mlp.down_proj                              98      0             -            -            -
  model.layers.58.mlp.gate_proj                              98      0             -            -            -
  model.layers.58.mlp.up_proj                                98      0             -            -            -
  model.layers.59.mlp.down_proj                              98      0             -            -            -
  model.layers.59.mlp.gate_proj                              98      0             -            -            -
  model.layers.59.mlp.up_proj                                98      0             -            -            -
  model.layers.59.self_attn.k_proj                           98      0             -            -            -
  model.layers.59.self_attn.o_proj                           98      0             -            -            -
  model.layers.59.self_attn.q_proj                           98      0             -            -            -
  model.layers.59.self_attn.v_proj                           98      0             -            -            -
  model.layers.6.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.6.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.6.linear_attn.out_proj                        98      0             -            -            -
  model.layers.6.mlp.down_proj                               98      0             -            -            -
  model.layers.6.mlp.gate_proj                               98      0             -            -            -
  model.layers.6.mlp.up_proj                                 98      0             -            -            -
  model.layers.60.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.60.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.60.linear_attn.out_proj                       98      0             -            -            -
  model.layers.60.mlp.down_proj                              98      0             -            -            -
  model.layers.60.mlp.gate_proj                              98      0             -            -            -
  model.layers.60.mlp.up_proj                                98      0             -            -            -
  model.layers.61.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.61.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.61.linear_attn.out_proj                       98      0             -            -            -
  model.layers.61.mlp.down_proj                              98      0             -            -            -
  model.layers.61.mlp.gate_proj                              98      0             -            -            -
  model.layers.61.mlp.up_proj                                98      0             -            -            -
  model.layers.62.linear_attn.in_proj_qkv                    98      0             -            -            -
  model.layers.62.linear_attn.in_proj_z                      98      0             -            -            -
  model.layers.62.linear_attn.out_proj                       98      0             -            -            -
  model.layers.62.mlp.down_proj                              98      0             -            -            -
  model.layers.62.mlp.gate_proj                              98      0             -            -            -
  model.layers.62.mlp.up_proj                                98      0             -            -            -
  model.layers.63.mlp.down_proj                              98      0             -            -            -
  model.layers.63.mlp.gate_proj                              98      0             -            -            -
  model.layers.63.mlp.up_proj                                98      0             -            -            -
  model.layers.63.self_attn.k_proj                           98      0             -            -            -
  model.layers.63.self_attn.o_proj                           98      0             -            -            -
  model.layers.63.self_attn.q_proj                           98      0             -            -            -
  model.layers.63.self_attn.v_proj                           98      0             -            -            -
  model.layers.7.mlp.down_proj                               98      0             -            -            -
  model.layers.7.mlp.gate_proj                               98      0             -            -            -
  model.layers.7.mlp.up_proj                                 98      0             -            -            -
  model.layers.7.self_attn.k_proj                            98      0             -            -            -
  model.layers.7.self_attn.o_proj                            98      0             -            -            -
  model.layers.7.self_attn.q_proj                            98      0             -            -            -
  model.layers.7.self_attn.v_proj                            98      0             -            -            -
  model.layers.8.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.8.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.8.linear_attn.out_proj                        98      0             -            -            -
  model.layers.8.mlp.down_proj                               98      0             -            -            -
  model.layers.8.mlp.gate_proj                               98      0             -            -            -
  model.layers.8.mlp.up_proj                                 98      0             -            -            -
  model.layers.9.linear_attn.in_proj_qkv                     98      0             -            -            -
  model.layers.9.linear_attn.in_proj_z                       98      0             -            -            -
  model.layers.9.linear_attn.out_proj                        98      0             -            -            -
  model.layers.9.mlp.down_proj                               98      0             -            -            -
  model.layers.9.mlp.gate_proj                               98      0             -            -            -
  model.layers.9.mlp.up_proj                                 98      0             -            -            -

THE TAIL over PREFILL positions only -- one per prefill chunk, so
every arm scores the SAME context. Decode positions are reported
separately and are NOT a numerics measure: once a token flips, the
arms are reading different contexts. The control proves it -- pooled
over decode, STD (which contains no int8 at all) has the same max KL
as INT8, so a pooled tail cannot attribute anything to the kernel.

arm            n        mean         p50         p95         p99         MAX     argmax flip
REF           48   0.000e+00   0.000e+00   0.000e+00   0.000e+00   0.000e+00 0/48 =   0.00%
INT8F         47   1.745e-01   6.352e-04   9.962e-01   5.182e+00   5.182e+00 3/47 =   6.38%
INT8FO        47   1.745e-01   6.352e-04   9.962e-01   5.182e+00   5.182e+00 3/47 =   6.38%

Decode positions (divergence-contaminated, shown for completeness):
arm            n        mean         MAX    greedy agree
REF           42   0.000e+00   0.000e+00 42/42 =  100.0%
INT8F         35   4.001e-02   1.343e+00 34/35 =   97.1%
INT8FO        35   4.001e-02   1.343e+00 34/35 =   97.1%

SAMPLER-RELEVANT TAIL @ canonical (T=1.0, top_p=0.95, top_k=20), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF           48     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 48     0.00%
INT8F         47      93.65%  6.657e-02  6.279e-01  9.579e-01          inf 16/ 47    10.64%
INT8FO        47      93.65%  6.657e-02  6.279e-01  9.579e-01          inf 16/ 47    10.64%

SAMPLER-RELEVANT TAIL @ owner's k~40 (T=1.0, top_p=1.0, top_k=40), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF           48     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/ 48     0.00%
INT8F         47      93.46%  6.520e-02  6.304e-01  9.474e-01          inf 39/ 47    11.17%
INT8FO        47      93.46%  6.520e-02  6.304e-01  9.474e-01          inf 39/ 47    11.17%

FULL-VOCAB numbers below, kept and labelled: they are the right
instrument for GREEDY or beam decode, where every rank matters.

arm         first-prefill-position KL       median          max
REF                         0.000e+00    0.000e+00    0.000e+00
INT8F                       3.712e-04    8.892e-05    1.826e-03
INT8FO                      3.712e-04    8.892e-05    1.826e-03

[i8fo] folded GEMMs=0  displaced (A fragments already clobbered)=0  unmatched (served the shipped way)=0
[i8f]  fused prep calls=39200  deferred-rotation flushes=0  unmatched=0
