[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (317092 > 262144). Running this sequence through the model will result in indexing errors
[i8] basket: 20 prompts x 16384 tok, domains ['code', 'longctx', 'math', 'sci', 'wiki']
compile-cache MISS: inductor-fx — torch.compile FX graphs + lowerings (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0]
compile-cache MISS: triton-cubin — triton kernel cubins (re)compiles this boot [/cache/arbi-serve/inductor-torch2.12.1+cu130-cu13.0-py3.12/u0/triton]
compile-cache MISS: tkv-autotune — tkv decode/prefill autotune tables (~min/shape) (re)compiles this boot [/cache/tkv]
compile-cache MISS: arbi-prefill-cute — arbi-prefill CuTeDSL kernel JIT (re)compiles this boot [/cache/cute-dsl]
compile-cache MISS: xgrammar — xgrammar compiled grammars (re)compiles this boot [/cache/xgrammar]
compile-cache MISS: megacache-fx — torch Mega-Cache bundle (Inductor FX + AOTAutograd backend) (re)compiles this boot [/cache/arbi-serve/compile-cache/588802f088df2611.bin]
Boot: loading model weights — still running (30s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
Boot: loading model weights — still running (60s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
Boot: loading model weights — still running (90s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
Boot: loading model weights — still running (120s). This phase reports nothing until it completes; it is progressing while this line advances. If it stops advancing, the phase is stuck.
default-pool residency snapshot returned no live ranges — the unpooled-weight fold cannot see what it would move
[TKV] Cold-boot kernel autotune starting: this is a real, expected one-time cost, not a hang — timing candidate decode-kernel configs for the shapes this deployment needs. Per-cell progress prints below as each one resolves; subsequent boots on this machine with an unchanged config reuse the cached table and skip this. See README.md's 'First boot on a new machine' section for real measured timings and how to skip this cost entirely with a pre-baked image (TKV_BAKE=1).
[TKV autotune] start: kv_cache shape=(1024, 256, 2048) data_ptr=660000000 in_capture=False (total_pages=1024) H_kv=4 num_sms=128 sw=0 buckets=(1024, 4096, 8192, 16384) batches=(1, 2) (capped at max_batch=1) reusable_cells=0 inherited=0 variants=['splitk'] tile_tokens=[4, 8, 16] min_blocks=[0, 3]
[TKV autotune] bucket=  1024 batch=  1: HELD splitk splits=32 tt=8 mb=0 (0.049ms) margin=0.166% sigma=0.367% z=1.09 n=2 inherited=False
[TKV autotune] bucket=  1024 batch=  2: HELD splitk splits=16 tt=16 mb=3 (0.048ms) margin=0.091% sigma=0.313% z=0.64 n=2 inherited=False
[TKV autotune] bucket=  4096 batch=  1: HELD splitk splits=64 tt=16 mb=3 (0.049ms) margin=0.625% sigma=0.434% z=3.42 n=2 inherited=False
[TKV autotune] bucket=  4096 batch=  2: HELD splitk splits=64 tt=16 mb=3 (0.048ms) margin=0.106% sigma=0.986% z=0.29 n=9 inherited=False
[TKV autotune] bucket=  8192 batch=  1: HELD splitk splits=128 tt=8 mb=0 (0.049ms) margin=0.007% sigma=1.267% z=0.02 n=11 inherited=False
[TKV autotune] bucket=  8192 batch=  2: HELD splitk splits=128 tt=8 mb=0 (0.049ms) margin=0.083% sigma=1.517% z=0.16 n=8 inherited=False
[TKV autotune] bucket= 16384 batch=  1: HELD splitk splits=256 tt=4 mb=3 (0.049ms) margin=0.365% sigma=2.680% z=0.40 n=11 inherited=False
[TKV autotune] bucket= 16384 batch=  2: HELD splitk splits=32 tt=4 mb=0 (0.153ms) margin=0.141% sigma=0.099% z=4.19 n=2 inherited=False
[TKV autotune] complete: 8 (batch, bucket) cells swept in 76.68s batch_invariant=0 tile_width_invariant=0
[TKV autotune-provenance] fp=57911e26f4e47f74 regime=swept cells=8 sweep_src=2772549a0ef713db tile_tokens=4,8,16 num_splits=16,32,64,128,256 min_blocks_per_sm=0,3 resolved=0 held=8 inherited=0
activation profile: the decode probe's FIRST call cost 8320171 allocator events / 1036 MiB / 77226.7 ms against 12102 / 1 MiB / 46.7 ms warm — a first-call kernel search inside the probe, not a step. The reported peak AND device time are the warm ones (2 passes, the last is what is reported); the cold numbers are logged so the difference is visible.
driver.modules_loaded baseline: SEEDED at 150994944 B (0.141 GiB) → /root/.cache/arbi-serve/budget-cache/84169d322e956268.modules.json. This boot ran UNGATED — it booked its own bracketed growth, so an unregistered pool or a raw cudaMalloc would be inside that number rather than on driver.residual. Every later boot at this configuration is held to it. Expected ONCE per configuration; if it repeats, the budget cache is not persisting (point ARBI_SERVE_BUDGET_CACHE_DIR at a durable volume) and the guard is inert.
/opt/venv/lib/python3.12/site-packages/torch/__init__.py:1172: FutureWarning: `torch.distributed.reduce_op` is deprecated, please use `torch.distributed.ReduceOp` instead
  return isinstance(obj, torch.Tensor)
/work/arbi_serve/engine/memory_budget/pool_residency.py:366: UserWarning: Accessing the data pointer of FakeTensor is deprecated and will error in PyTorch 2.5. This is almost definitely a bug in your code and will cause undefined behavior with subsystems like torch.compile. Please wrap calls to tensor.data_ptr() in an opaque custom op; If all else fails, you can guard accesses to tensor.data_ptr() on isinstance(tensor, FakeTensor). (Triggered internally at /pytorch/c10/core/StorageImpl.cpp:34.)
  if not _in_default_pool(obj.data_ptr()):
[i8] trellis->path map: 401 linears
JIT compile AFTER serving-ready [cpp_ext]: exl3_i8_gemm_k4 cached .so load — a live request paid this compile's latency. This is a boot-warmup coverage gap: extend warmup to pre-compile this kernel/specialization. Counter jit_compile_serving (must-not-fire) at GET /v1/admin/flag_truth.
[i8] prefix cache: server default resolved=True, per-request cache_enabled=False; any cache hit below is refused
activation admission RE-ARMED (JIT compile after serving-ready): the step budget FELL to 5260 MiB, -2 MiB on the 5262 MiB it was enforcing and 5262 MiB the boot reading armed. The widest slate admission will build is narrower from here — narrower prefill chunks and deferred rows instead of a forward that OOMs after the layout freeze.
driver serving-growth OVER RESERVE: the driver holds 2 MiB more than it did when the boot brackets closed, against the 0 MiB the serving floor's transient.serving_step.driver_growth term held for it. Those bytes are in no pool and no allocator counter -- the live card reaches them as driver.modules_serving and the boot ledger only as driver.residual -- and the KV layout is frozen, so the overage comes out of the free VRAM the floor is holding for one in-flight step (the verify tail, the DFlash context-assemble) and those reserves are the ones that fail first. The reading is on record for this configuration, so the next boot holds it; this process serves the rest of its life short by the overage.
[i8] warmup done
  [0] code           REF      16384tok  wall=  5.17s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [0] code           REF2     16384tok  wall=  5.15s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [0] code           STD      16384tok  wall=  5.43s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           INT8     16384tok  wall=  4.92s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           INT8G    16384tok  wall=  6.15s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           INT8G256  16384tok  wall=  5.83s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           INT8GN   16384tok  wall=  5.93s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           SIMROW   16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           SIMG128  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           SIMG256  16384tok  wall=  7.68s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [0] code           SIMG512  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           REF      16384tok  wall=  5.18s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [1] code           REF2     16384tok  wall=  5.20s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [1] code           STD      16384tok  wall=  5.49s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           INT8G    16384tok  wall=  6.17s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           INT8G256  16384tok  wall=  5.86s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           INT8GN   16384tok  wall=  5.97s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           SIMROW   16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           SIMG128  16384tok  wall=  7.71s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           SIMG256  16384tok  wall=  7.69s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [1] code           SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           REF      16384tok  wall=  5.20s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [2] code           REF2     16384tok  wall=  5.20s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [2] code           STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           INT8     16384tok  wall=  4.94s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           SIMROW   16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           SIMG128  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           SIMG256  16384tok  wall=  7.71s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [2] code           SIMG512  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [3] code           REF2     16384tok  wall=  5.20s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [3] code           STD      16384tok  wall=  5.49s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           INT8     16384tok  wall=  4.94s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           INT8G    16384tok  wall=  6.18s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           SIMROW   16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           SIMG256  16384tok  wall=  7.71s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [3] code           SIMG512  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           REF      16384tok  wall=  5.21s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [4] wiki           REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [4] wiki           STD      16384tok  wall=  5.49s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           INT8     16384tok  wall=  4.94s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           INT8G    16384tok  wall=  6.18s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           SIMG256  16384tok  wall=  7.70s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [4] wiki           SIMG512  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           REF      16384tok  wall=  5.21s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [5] wiki           REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [5] wiki           STD      16384tok  wall=  5.49s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           INT8G    16384tok  wall=  6.18s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           SIMG256  16384tok  wall=  7.71s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [5] wiki           SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           REF      16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [6] wiki           REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [6] wiki           STD      16384tok  wall=  5.51s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           INT8G256  16384tok  wall=  5.89s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           SIMROW   16384tok  wall=  7.77s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           SIMG256  16384tok  wall=  7.71s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [6] wiki           SIMG512  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           REF      16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [7] wiki           REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [7] wiki           STD      16384tok  wall=  5.52s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           INT8G    16384tok  wall=  6.19s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           INT8G256  16384tok  wall=  5.88s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           SIMG256  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [7] wiki           SIMG512  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [8] math           REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [8] math           STD      16384tok  wall=  5.51s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           INT8G256  16384tok  wall=  5.89s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           SIMG128  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [8] math           SIMG512  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [9] math           REF2     16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [9] math           STD      16384tok  wall=  5.52s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           INT8G256  16384tok  wall=  5.89s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [9] math           SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [10] math           REF2     16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [10] math           STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           INT8G    16384tok  wall=  6.21s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           INT8G256  16384tok  wall=  5.89s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           SIMG256  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [10] math           SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           REF      16384tok  wall=  5.25s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [11] math           REF2     16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [11] math           STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           INT8G    16384tok  wall=  6.21s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           INT8G256  16384tok  wall=  5.88s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           SIMG128  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           SIMG256  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [11] math           SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [12] longctx        REF2     16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [12] longctx        STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        INT8G    16384tok  wall=  6.19s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        INT8G256  16384tok  wall=  5.88s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        INT8GN   16384tok  wall=  5.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        SIMROW   16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        SIMG128  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        SIMG256  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [12] longctx        SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [13] longctx        REF2     16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [13] longctx        STD      16384tok  wall=  5.51s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        INT8G256  16384tok  wall=  5.88s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        SIMG128  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        SIMG256  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [13] longctx        SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [14] longctx        REF2     16384tok  wall=  5.25s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [14] longctx        STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        INT8     16384tok  wall=  4.95s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        SIMG128  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [14] longctx        SIMG512  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [15] longctx        REF2     16384tok  wall=  5.25s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [15] longctx        STD      16384tok  wall=  5.51s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        INT8G256  16384tok  wall=  5.87s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        INT8GN   16384tok  wall=  5.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        SIMG128  16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [15] longctx        SIMG512  16384tok  wall=  7.75s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [16] sci            REF2     16384tok  wall=  5.26s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [16] sci            STD      16384tok  wall=  5.50s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            INT8     16384tok  wall=  5.26s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            INT8G    16384tok  wall=  6.22s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            INT8G256  16384tok  wall=  5.91s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            INT8GN   16384tok  wall=  5.98s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [16] sci            SIMG512  16384tok  wall=  7.79s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [17] sci            REF2     16384tok  wall=  5.22s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [17] sci            STD      16384tok  wall=  5.52s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            INT8     16384tok  wall=  4.96s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            INT8G    16384tok  wall=  6.21s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            INT8G256  16384tok  wall=  5.90s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            INT8GN   16384tok  wall=  5.97s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            SIMG256  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [17] sci            SIMG512  16384tok  wall=  7.77s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            REF      16384tok  wall=  5.23s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [18] sci            REF2     16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [18] sci            STD      16384tok  wall=  5.52s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            INT8     16384tok  wall=  4.97s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            INT8G    16384tok  wall=  6.21s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            INT8G256  16384tok  wall=  5.91s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            INT8GN   16384tok  wall=  5.97s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            SIMROW   16384tok  wall=  7.76s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            SIMG256  16384tok  wall=  7.72s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [18] sci            SIMG512  16384tok  wall=  7.78s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            REF      16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [19] sci            REF2     16384tok  wall=  5.24s  finish=length  leg=3200F/0S acc=3200x16/0x32
  [19] sci            STD      16384tok  wall=  5.52s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            INT8     16384tok  wall=  4.98s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            INT8G    16384tok  wall=  6.20s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            INT8G256  16384tok  wall=  5.90s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            INT8GN   16384tok  wall=  5.98s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            SIMROW   16384tok  wall=  7.78s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            SIMG128  16384tok  wall=  7.74s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            SIMG256  16384tok  wall=  7.73s  finish=length  leg=0F/3200S acc=3200x16/0x32
  [19] sci            SIMG512  16384tok  wall=  7.78s  finish=length  leg=0F/3200S acc=3200x16/0x32

[i8] kernel census: int8 calls=256000 rows=524288000 reconstructs SKIPPED=256000 fallback (non-4bpw) calls=0
[i8] shapes served by the kernel (K,N)->calls: {(5120, 1024): 20480, (5120, 6144): 30720, (5120, 10240): 30720, (5120, 12288): 10240, (5120, 17408): 81920, (6144, 5120): 40960, (17408, 5120): 40960}
[i8] excluded projections: none  excluded calls=0

  linear class                                           served   excl  int8 vs fp32 legB vs fp32 int8 vs legB
  model.layers.0.linear_attn.in_proj_qkv                    644      0     4.827e-03    1.713e-03    5.132e-03
  model.layers.0.linear_attn.in_proj_z                      644      0     6.316e-03    1.129e-03    6.418e-03
  model.layers.0.linear_attn.out_proj                       644      0     1.249e-02    2.189e-03    1.268e-02
  model.layers.0.mlp.down_proj                              644      0     2.123e-02    2.610e-03    2.139e-02
  model.layers.0.mlp.gate_proj                              644      0     9.121e-03    3.416e-03    9.741e-03
  model.layers.0.mlp.up_proj                                644      0     9.089e-03    3.416e-03    9.708e-03
  model.layers.1.linear_attn.in_proj_qkv                    644      0     1.187e-02    2.166e-03    1.207e-02
  model.layers.1.linear_attn.in_proj_z                      644      0     1.361e-02    1.840e-03    1.374e-02
  model.layers.1.linear_attn.out_proj                       644      0     1.607e-02    2.259e-03    1.622e-02
  model.layers.1.mlp.down_proj                              644      0     1.342e-02    2.320e-03    1.362e-02
  model.layers.1.mlp.gate_proj                              644      0     9.697e-03    3.460e-03    1.030e-02
  model.layers.1.mlp.up_proj                                644      0     1.014e-02    3.489e-03    1.072e-02
  model.layers.10.linear_attn.in_proj_qkv                   644      0     1.305e-02    2.252e-03    1.324e-02
  model.layers.10.linear_attn.in_proj_z                     644      0     1.299e-02    1.839e-03    1.312e-02
  model.layers.10.linear_attn.out_proj                      644      0     1.497e-02    2.122e-03    1.512e-02
  model.layers.10.mlp.down_proj                             644      0     1.269e-02    2.344e-03    1.291e-02
  model.layers.10.mlp.gate_proj                             644      0     1.035e-02    3.580e-03    1.095e-02
  model.layers.10.mlp.up_proj                               644      0     1.075e-02    3.590e-03    1.134e-02
  model.layers.11.mlp.down_proj                             644      0     1.334e-02    2.506e-03    1.357e-02
  model.layers.11.mlp.gate_proj                             644      0     1.027e-02    3.508e-03    1.086e-02
  model.layers.11.mlp.up_proj                               644      0     1.088e-02    3.578e-03    1.145e-02
  model.layers.11.self_attn.k_proj                          644      0     1.328e-02    2.698e-03    1.356e-02
  model.layers.11.self_attn.o_proj                          644      0     1.304e-02    2.371e-03    1.326e-02
  model.layers.11.self_attn.q_proj                          644      0     1.174e-02    2.023e-03    1.192e-02
  model.layers.11.self_attn.v_proj                          644      0     1.037e-02    2.285e-03    1.062e-02
  model.layers.12.linear_attn.in_proj_qkv                   644      0     1.293e-02    2.244e-03    1.312e-02
  model.layers.12.linear_attn.in_proj_z                     644      0     1.367e-02    1.892e-03    1.380e-02
  model.layers.12.linear_attn.out_proj                      644      0     1.581e-02    2.140e-03    1.596e-02
  model.layers.12.mlp.down_proj                             644      0     1.317e-02    2.548e-03    1.341e-02
  model.layers.12.mlp.gate_proj                             644      0     1.002e-02    3.476e-03    1.061e-02
  model.layers.12.mlp.up_proj                               644      0     1.072e-02    3.553e-03    1.130e-02
  model.layers.13.linear_attn.in_proj_qkv                   644      0     1.292e-02    2.242e-03    1.312e-02
  model.layers.13.linear_attn.in_proj_z                     644      0     1.418e-02    1.977e-03    1.432e-02
  model.layers.13.linear_attn.out_proj                      644      0     1.680e-02    2.178e-03    1.695e-02
  model.layers.13.mlp.down_proj                             644      0     1.372e-02    2.565e-03    1.396e-02
  model.layers.13.mlp.gate_proj                             644      0     1.007e-02    3.482e-03    1.065e-02
  model.layers.13.mlp.up_proj                               644      0     1.070e-02    3.556e-03    1.128e-02
  model.layers.14.linear_attn.in_proj_qkv                   644      0     1.270e-02    2.232e-03    1.289e-02
  model.layers.14.linear_attn.in_proj_z                     644      0     1.336e-02    1.881e-03    1.349e-02
  model.layers.14.linear_attn.out_proj                      644      0     1.588e-02    2.213e-03    1.603e-02
  model.layers.14.mlp.down_proj                             644      0     1.300e-02    2.462e-03    1.324e-02
  model.layers.14.mlp.gate_proj                             644      0     9.965e-03    3.501e-03    1.056e-02
  model.layers.14.mlp.up_proj                               644      0     1.064e-02    3.564e-03    1.122e-02
  model.layers.15.mlp.down_proj                             644      0     1.349e-02    2.531e-03    1.373e-02
  model.layers.15.mlp.gate_proj                             644      0     1.010e-02    3.486e-03    1.068e-02
  model.layers.15.mlp.up_proj                               644      0     1.067e-02    3.562e-03    1.125e-02
  model.layers.15.self_attn.k_proj                          644      0     1.455e-02    2.891e-03    1.483e-02
  model.layers.15.self_attn.o_proj                          644      0     1.396e-02    2.117e-03    1.412e-02
  model.layers.15.self_attn.q_proj                          644      0     1.301e-02    2.088e-03    1.318e-02
  model.layers.15.self_attn.v_proj                          644      0     9.798e-03    2.225e-03    1.005e-02
  model.layers.16.linear_attn.in_proj_qkv                   644      0     1.287e-02    2.243e-03    1.307e-02
  model.layers.16.linear_attn.in_proj_z                     644      0     1.354e-02    1.892e-03    1.367e-02
  model.layers.16.linear_attn.out_proj                      644      0     1.636e-02    2.280e-03    1.652e-02
  model.layers.16.mlp.down_proj                             644      0     1.296e-02    2.518e-03    1.321e-02
  model.layers.16.mlp.gate_proj                             644      0     1.005e-02    3.475e-03    1.064e-02
  model.layers.16.mlp.up_proj                               644      0     1.071e-02    3.559e-03    1.128e-02
  model.layers.17.linear_attn.in_proj_qkv                   644      0     1.287e-02    2.223e-03    1.306e-02
  model.layers.17.linear_attn.in_proj_z                     644      0     1.350e-02    1.895e-03    1.364e-02
  model.layers.17.linear_attn.out_proj                      644      0     1.778e-02    2.246e-03    1.792e-02
  model.layers.17.mlp.down_proj                             644      0     1.381e-02    2.456e-03    1.402e-02
  model.layers.17.mlp.gate_proj                             644      0     1.010e-02    3.469e-03    1.068e-02
  model.layers.17.mlp.up_proj                               644      0     1.072e-02    3.565e-03    1.129e-02
  model.layers.18.linear_attn.in_proj_qkv                   644      0     1.206e-02    2.188e-03    1.226e-02
  model.layers.18.linear_attn.in_proj_z                     644      0     1.264e-02    1.787e-03    1.276e-02
  model.layers.18.linear_attn.out_proj                      644      0     1.529e-02    2.227e-03    1.545e-02
  model.layers.18.mlp.down_proj                             644      0     1.459e-02    2.424e-03    1.479e-02
  model.layers.18.mlp.gate_proj                             644      0     1.024e-02    3.424e-03    1.080e-02
  model.layers.18.mlp.up_proj                               644      0     1.058e-02    3.531e-03    1.115e-02
  model.layers.19.mlp.down_proj                             644      0     1.586e-02    2.536e-03    1.606e-02
  model.layers.19.mlp.gate_proj                             644      0     1.007e-02    3.397e-03    1.063e-02
  model.layers.19.mlp.up_proj                               644      0     1.096e-02    3.579e-03    1.154e-02
  model.layers.19.self_attn.k_proj                          644      0     1.406e-02    2.789e-03    1.434e-02
  model.layers.19.self_attn.o_proj                          644      0     1.812e-02    2.164e-03    1.825e-02
  model.layers.19.self_attn.q_proj                          644      0     9.499e-03    1.928e-03    9.693e-03
  model.layers.19.self_attn.v_proj                          644      0     1.047e-02    2.344e-03    1.072e-02
  model.layers.2.linear_attn.in_proj_qkv                    644      0     1.249e-02    2.214e-03    1.269e-02
  model.layers.2.linear_attn.in_proj_z                      644      0     1.398e-02    1.925e-03    1.412e-02
  model.layers.2.linear_attn.out_proj                       644      0     1.798e-02    2.323e-03    1.812e-02
  model.layers.2.mlp.down_proj                              644      0     1.704e-02    2.374e-03    1.721e-02
  model.layers.2.mlp.gate_proj                              644      0     1.030e-02    3.532e-03    1.089e-02
  model.layers.2.mlp.up_proj                                644      0     1.062e-02    3.557e-03    1.120e-02
  model.layers.20.linear_attn.in_proj_qkv                   644      0     1.213e-02    2.136e-03    1.232e-02
  model.layers.20.linear_attn.in_proj_z                     644      0     1.273e-02    1.792e-03    1.285e-02
  model.layers.20.linear_attn.out_proj                      644      0     1.582e-02    2.229e-03    1.597e-02
  model.layers.20.mlp.down_proj                             644      0     1.593e-02    2.584e-03    1.614e-02
  model.layers.20.mlp.gate_proj                             644      0     1.008e-02    3.405e-03    1.064e-02
  model.layers.20.mlp.up_proj                               644      0     1.127e-02    3.606e-03    1.183e-02
  model.layers.21.linear_attn.in_proj_qkv                   644      0     1.198e-02    2.198e-03    1.218e-02
  model.layers.21.linear_attn.in_proj_z                     644      0     1.161e-02    1.671e-03    1.173e-02
  model.layers.21.linear_attn.out_proj                      644      0     1.699e-02    2.233e-03    1.713e-02
  model.layers.21.mlp.down_proj                             644      0     1.720e-02    2.590e-03    1.739e-02
  model.layers.21.mlp.gate_proj                             644      0     1.010e-02    3.421e-03    1.066e-02
  model.layers.21.mlp.up_proj                               644      0     1.123e-02    3.605e-03    1.180e-02
  model.layers.22.linear_attn.in_proj_qkv                   644      0     1.201e-02    2.236e-03    1.222e-02
  model.layers.22.linear_attn.in_proj_z                     644      0     1.203e-02    1.713e-03    1.215e-02
  model.layers.22.linear_attn.out_proj                      644      0     1.578e-02    2.301e-03    1.595e-02
  model.layers.22.mlp.down_proj                             644      0     1.514e-02    2.549e-03    1.536e-02
  model.layers.22.mlp.gate_proj                             644      0     1.053e-02    3.139e-03    1.099e-02
  model.layers.22.mlp.up_proj                               644      0     1.141e-02    3.619e-03    1.197e-02
  model.layers.23.mlp.down_proj                             644      0     1.569e-02    2.554e-03    1.589e-02
  model.layers.23.mlp.gate_proj                             644      0     1.009e-02    3.372e-03    1.064e-02
  model.layers.23.mlp.up_proj                               644      0     1.141e-02    3.621e-03    1.197e-02
  model.layers.23.self_attn.k_proj                          644      0     1.378e-02    2.666e-03    1.404e-02
  model.layers.23.self_attn.o_proj                          644      0     1.681e-02    2.421e-03    1.698e-02
  model.layers.23.self_attn.q_proj                          644      0     9.469e-03    1.965e-03    9.672e-03
  model.layers.23.self_attn.v_proj                          644      0     1.034e-02    2.328e-03    1.059e-02
  model.layers.24.linear_attn.in_proj_qkv                   644      0     1.188e-02    2.254e-03    1.209e-02
  model.layers.24.linear_attn.in_proj_z                     644      0     1.222e-02    1.731e-03    1.234e-02
  model.layers.24.linear_attn.out_proj                      644      0     1.606e-02    2.272e-03    1.622e-02
  model.layers.24.mlp.down_proj                             644      0     1.456e-02    2.622e-03    1.480e-02
  model.layers.24.mlp.gate_proj                             644      0     1.027e-02    3.317e-03    1.079e-02
  model.layers.24.mlp.up_proj                               644      0     1.144e-02    3.614e-03    1.200e-02
  model.layers.25.linear_attn.in_proj_qkv                   644      0     1.142e-02    2.213e-03    1.163e-02
  model.layers.25.linear_attn.in_proj_z                     644      0     1.278e-02    1.790e-03    1.291e-02
  model.layers.25.linear_attn.out_proj                      644      0     1.473e-02    2.175e-03    1.489e-02
  model.layers.25.mlp.down_proj                             644      0     1.392e-02    2.551e-03    1.415e-02
  model.layers.25.mlp.gate_proj                             644      0     1.054e-02    3.324e-03    1.105e-02
  model.layers.25.mlp.up_proj                               644      0     1.130e-02    3.600e-03    1.186e-02
  model.layers.26.linear_attn.in_proj_qkv                   644      0     1.149e-02    2.236e-03    1.171e-02
  model.layers.26.linear_attn.in_proj_z                     644      0     1.258e-02    1.797e-03    1.271e-02
  model.layers.26.linear_attn.out_proj                      644      0     1.493e-02    2.343e-03    1.511e-02
  model.layers.26.mlp.down_proj                             644      0     1.270e-02    2.467e-03    1.294e-02
  model.layers.26.mlp.gate_proj                             644      0     1.111e-02    3.295e-03    1.159e-02
  model.layers.26.mlp.up_proj                               644      0     1.086e-02    3.547e-03    1.142e-02
  model.layers.27.mlp.down_proj                             644      0     1.336e-02    2.576e-03    1.360e-02
  model.layers.27.mlp.gate_proj                             644      0     1.080e-02    3.203e-03    1.126e-02
  model.layers.27.mlp.up_proj                               644      0     1.090e-02    3.553e-03    1.146e-02
  model.layers.27.self_attn.k_proj                          644      0     1.216e-02    2.498e-03    1.241e-02
  model.layers.27.self_attn.o_proj                          644      0     1.711e-02    2.745e-03    1.732e-02
  model.layers.27.self_attn.q_proj                          644      0     1.021e-02    2.061e-03    1.041e-02
  model.layers.27.self_attn.v_proj                          644      0     9.822e-03    2.244e-03    1.008e-02
  model.layers.28.linear_attn.in_proj_qkv                   644      0     1.178e-02    2.221e-03    1.199e-02
  model.layers.28.linear_attn.in_proj_z                     644      0     1.288e-02    1.818e-03    1.301e-02
  model.layers.28.linear_attn.out_proj                      644      0     1.477e-02    2.295e-03    1.495e-02
  model.layers.28.mlp.down_proj                             644      0     1.301e-02    2.596e-03    1.326e-02
  model.layers.28.mlp.gate_proj                             644      0     1.103e-02    3.174e-03    1.148e-02
  model.layers.28.mlp.up_proj                               644      0     1.084e-02    3.538e-03    1.140e-02
  model.layers.29.linear_attn.in_proj_qkv                   644      0     1.204e-02    2.244e-03    1.224e-02
  model.layers.29.linear_attn.in_proj_z                     644      0     1.396e-02    1.962e-03    1.409e-02
  model.layers.29.linear_attn.out_proj                      644      0     1.654e-02    2.351e-03    1.670e-02
  model.layers.29.mlp.down_proj                             644      0     1.347e-02    2.622e-03    1.372e-02
  model.layers.29.mlp.gate_proj                             644      0     1.114e-02    3.185e-03    1.159e-02
  model.layers.29.mlp.up_proj                               644      0     1.080e-02    3.537e-03    1.136e-02
  model.layers.3.mlp.down_proj                              644      0     1.419e-02    2.578e-03    1.442e-02
  model.layers.3.mlp.gate_proj                              644      0     1.018e-02    3.522e-03    1.077e-02
  model.layers.3.mlp.up_proj                                644      0     1.100e-02    3.600e-03    1.157e-02
  model.layers.3.self_attn.k_proj                           644      0     1.246e-02    2.588e-03    1.273e-02
  model.layers.3.self_attn.o_proj                           644      0     1.494e-02    2.175e-03    1.510e-02
  model.layers.3.self_attn.q_proj                           644      0     1.169e-02    2.036e-03    1.186e-02
  model.layers.3.self_attn.v_proj                           644      0     1.263e-02    2.305e-03    1.284e-02
  model.layers.30.linear_attn.in_proj_qkv                   644      0     1.190e-02    2.230e-03    1.211e-02
  model.layers.30.linear_attn.in_proj_z                     644      0     1.350e-02    1.891e-03    1.363e-02
  model.layers.30.linear_attn.out_proj                      644      0     1.623e-02    2.415e-03    1.641e-02
  model.layers.30.mlp.down_proj                             644      0     1.285e-02    2.560e-03    1.310e-02
  model.layers.30.mlp.gate_proj                             644      0     1.172e-02    3.189e-03    1.214e-02
  model.layers.30.mlp.up_proj                               644      0     1.067e-02    3.520e-03    1.124e-02
  model.layers.31.mlp.down_proj                             644      0     1.318e-02    2.599e-03    1.343e-02
  model.layers.31.mlp.gate_proj                             644      0     1.148e-02    3.157e-03    1.190e-02
  model.layers.31.mlp.up_proj                               644      0     1.089e-02    3.517e-03    1.145e-02
  model.layers.31.self_attn.k_proj                          644      0     1.206e-02    2.514e-03    1.232e-02
  model.layers.31.self_attn.o_proj                          644      0     2.216e-02    3.029e-03    2.236e-02
  model.layers.31.self_attn.q_proj                          644      0     1.017e-02    2.075e-03    1.038e-02
  model.layers.31.self_attn.v_proj                          644      0     9.911e-03    2.190e-03    1.015e-02
  model.layers.32.linear_attn.in_proj_qkv                   644      0     1.281e-02    2.287e-03    1.301e-02
  model.layers.32.linear_attn.in_proj_z                     644      0     1.411e-02    1.984e-03    1.424e-02
  model.layers.32.linear_attn.out_proj                      644      0     1.565e-02    2.305e-03    1.582e-02
  model.layers.32.mlp.down_proj                             644      0     1.303e-02    2.616e-03    1.329e-02
  model.layers.32.mlp.gate_proj                             644      0     1.165e-02    3.105e-03    1.206e-02
  model.layers.32.mlp.up_proj                               644      0     1.107e-02    3.522e-03    1.162e-02
  model.layers.33.linear_attn.in_proj_qkv                   644      0     1.260e-02    2.297e-03    1.280e-02
  model.layers.33.linear_attn.in_proj_z                     644      0     1.385e-02    1.989e-03    1.400e-02
  model.layers.33.linear_attn.out_proj                      644      0     1.744e-02    2.367e-03    1.760e-02
  model.layers.33.mlp.down_proj                             644      0     1.341e-02    2.604e-03    1.366e-02
  model.layers.33.mlp.gate_proj                             644      0     1.192e-02    3.114e-03    1.232e-02
  model.layers.33.mlp.up_proj                               644      0     1.124e-02    3.539e-03    1.178e-02
  model.layers.34.linear_attn.in_proj_qkv                   644      0     1.252e-02    2.275e-03    1.272e-02
  model.layers.34.linear_attn.in_proj_z                     644      0     1.343e-02    1.914e-03    1.356e-02
  model.layers.34.linear_attn.out_proj                      644      0     1.613e-02    2.365e-03    1.630e-02
  model.layers.34.mlp.down_proj                             644      0     1.377e-02    2.611e-03    1.402e-02
  model.layers.34.mlp.gate_proj                             644      0     1.207e-02    3.072e-03    1.245e-02
  model.layers.34.mlp.up_proj                               644      0     1.115e-02    3.508e-03    1.169e-02
  model.layers.35.mlp.down_proj                             644      0     1.414e-02    2.562e-03    1.437e-02
  model.layers.35.mlp.gate_proj                             644      0     1.065e-02    3.066e-03    1.108e-02
  model.layers.35.mlp.up_proj                               644      0     1.138e-02    3.581e-03    1.193e-02
  model.layers.35.self_attn.k_proj                          644      0     1.174e-02    2.499e-03    1.201e-02
  model.layers.35.self_attn.o_proj                          644      0     1.996e-02    2.805e-03    2.016e-02
  model.layers.35.self_attn.q_proj                          644      0     1.073e-02    2.114e-03    1.093e-02
  model.layers.35.self_attn.v_proj                          644      0     1.069e-02    2.260e-03    1.092e-02
  model.layers.36.linear_attn.in_proj_qkv                   644      0     1.293e-02    2.275e-03    1.314e-02
  model.layers.36.linear_attn.in_proj_z                     644      0     1.209e-02    1.744e-03    1.221e-02
  model.layers.36.linear_attn.out_proj                      644      0     1.664e-02    2.260e-03    1.679e-02
  model.layers.36.mlp.down_proj                             644      0     1.511e-02    2.625e-03    1.534e-02
  model.layers.36.mlp.gate_proj                             644      0     1.026e-02    3.146e-03    1.073e-02
  model.layers.36.mlp.up_proj                               644      0     1.153e-02    3.605e-03    1.208e-02
  model.layers.37.linear_attn.in_proj_qkv                   644      0     1.222e-02    2.241e-03    1.242e-02
  model.layers.37.linear_attn.in_proj_z                     644      0     1.057e-02    1.598e-03    1.069e-02
  model.layers.37.linear_attn.out_proj                      644      0     1.841e-02    2.196e-03    1.854e-02
  model.layers.37.mlp.down_proj                             644      0     1.541e-02    2.734e-03    1.565e-02
  model.layers.37.mlp.gate_proj                             644      0     1.056e-02    3.236e-03    1.104e-02
  model.layers.37.mlp.up_proj                               644      0     1.173e-02    3.629e-03    1.228e-02
  model.layers.38.linear_attn.in_proj_qkv                   644      0     1.201e-02    2.319e-03    1.223e-02
  model.layers.38.linear_attn.in_proj_z                     644      0     1.100e-02    1.629e-03    1.112e-02
  model.layers.38.linear_attn.out_proj                      644      0     1.659e-02    2.312e-03    1.675e-02
  model.layers.38.mlp.down_proj                             644      0     1.600e-02    2.698e-03    1.622e-02
  model.layers.38.mlp.gate_proj                             644      0     1.070e-02    3.068e-03    1.113e-02
  model.layers.38.mlp.up_proj                               644      0     1.171e-02    3.642e-03    1.227e-02
  model.layers.39.mlp.down_proj                             644      0     1.436e-02    2.656e-03    1.461e-02
  model.layers.39.mlp.gate_proj                             644      0     9.969e-03    3.158e-03    1.046e-02
  model.layers.39.mlp.up_proj                               644      0     1.180e-02    3.648e-03    1.235e-02
  model.layers.39.self_attn.k_proj                          644      0     1.179e-02    2.469e-03    1.205e-02
  model.layers.39.self_attn.o_proj                          644      0     2.073e-02    2.839e-03    2.093e-02
  model.layers.39.self_attn.q_proj                          644      0     1.046e-02    2.095e-03    1.066e-02
  model.layers.39.self_attn.v_proj                          644      0     1.092e-02    2.333e-03    1.116e-02
  model.layers.4.linear_attn.in_proj_qkv                    644      0     1.192e-02    2.308e-03    1.214e-02
  model.layers.4.linear_attn.in_proj_z                      644      0     1.245e-02    1.774e-03    1.258e-02
  model.layers.4.linear_attn.out_proj                       644      0     1.792e-02    2.410e-03    1.808e-02
  model.layers.4.mlp.down_proj                              644      0     1.444e-02    2.608e-03    1.467e-02
  model.layers.4.mlp.gate_proj                              644      0     1.014e-02    3.510e-03    1.073e-02
  model.layers.4.mlp.up_proj                                644      0     1.122e-02    3.616e-03    1.179e-02
  model.layers.40.linear_attn.in_proj_qkv                   644      0     1.225e-02    2.282e-03    1.246e-02
  model.layers.40.linear_attn.in_proj_z                     644      0     1.172e-02    1.687e-03    1.184e-02
  model.layers.40.linear_attn.out_proj                      644      0     1.605e-02    2.268e-03    1.621e-02
  model.layers.40.mlp.down_proj                             644      0     1.482e-02    2.678e-03    1.506e-02
  model.layers.40.mlp.gate_proj                             644      0     1.009e-02    3.135e-03    1.056e-02
  model.layers.40.mlp.up_proj                               644      0     1.164e-02    3.642e-03    1.220e-02
  model.layers.41.linear_attn.in_proj_qkv                   644      0     1.176e-02    2.259e-03    1.198e-02
  model.layers.41.linear_attn.in_proj_z                     644      0     1.224e-02    1.731e-03    1.237e-02
  model.layers.41.linear_attn.out_proj                      644      0     1.650e-02    2.256e-03    1.666e-02
  model.layers.41.mlp.down_proj                             644      0     1.479e-02    2.701e-03    1.504e-02
  model.layers.41.mlp.gate_proj                             644      0     1.085e-02    3.215e-03    1.132e-02
  model.layers.41.mlp.up_proj                               644      0     1.165e-02    3.636e-03    1.220e-02
  model.layers.42.linear_attn.in_proj_qkv                   644      0     1.163e-02    2.276e-03    1.185e-02
  model.layers.42.linear_attn.in_proj_z                     644      0     1.242e-02    1.746e-03    1.254e-02
  model.layers.42.linear_attn.out_proj                      644      0     1.567e-02    2.282e-03    1.583e-02
  model.layers.42.mlp.down_proj                             644      0     1.453e-02    2.664e-03    1.477e-02
  model.layers.42.mlp.gate_proj                             644      0     1.173e-02    3.239e-03    1.217e-02
  model.layers.42.mlp.up_proj                               644      0     1.142e-02    3.582e-03    1.197e-02
  model.layers.43.mlp.down_proj                             644      0     1.385e-02    2.671e-03    1.411e-02
  model.layers.43.mlp.gate_proj                             644      0     1.084e-02    3.217e-03    1.131e-02
  model.layers.43.mlp.up_proj                               644      0     1.150e-02    3.615e-03    1.206e-02
  model.layers.43.self_attn.k_proj                          644      0     1.246e-02    2.466e-03    1.271e-02
  model.layers.43.self_attn.o_proj                          644      0     2.163e-02    3.132e-03    2.185e-02
  model.layers.43.self_attn.q_proj                          644      0     1.058e-02    2.089e-03    1.078e-02
  model.layers.43.self_attn.v_proj                          644      0     1.044e-02    2.263e-03    1.069e-02
  model.layers.44.linear_attn.in_proj_qkv                   644      0     1.204e-02    2.230e-03    1.225e-02
  model.layers.44.linear_attn.in_proj_z                     644      0     1.224e-02    1.700e-03    1.236e-02
  model.layers.44.linear_attn.out_proj                      644      0     1.626e-02    2.298e-03    1.643e-02
  model.layers.44.mlp.down_proj                             644      0     1.413e-02    2.713e-03    1.439e-02
  model.layers.44.mlp.gate_proj                             644      0     1.083e-02    3.261e-03    1.131e-02
  model.layers.44.mlp.up_proj                               644      0     1.171e-02    3.618e-03    1.225e-02
  model.layers.45.linear_attn.in_proj_qkv                   644      0     1.207e-02    2.277e-03    1.228e-02
  model.layers.45.linear_attn.in_proj_z                     644      0     1.264e-02    1.763e-03    1.276e-02
  model.layers.45.linear_attn.out_proj                      644      0     1.812e-02    2.434e-03    1.828e-02
  model.layers.45.mlp.down_proj                             644      0     1.437e-02    2.742e-03    1.463e-02
  model.layers.45.mlp.gate_proj                             644      0     1.082e-02    3.287e-03    1.131e-02
  model.layers.45.mlp.up_proj                               644      0     1.155e-02    3.622e-03    1.211e-02
  model.layers.46.linear_attn.in_proj_qkv                   644      0     1.195e-02    2.245e-03    1.216e-02
  model.layers.46.linear_attn.in_proj_z                     644      0     1.186e-02    1.653e-03    1.197e-02
  model.layers.46.linear_attn.out_proj                      644      0     1.869e-02    2.371e-03    1.884e-02
  model.layers.46.mlp.down_proj                             644      0     1.422e-02    2.687e-03    1.447e-02
  model.layers.46.mlp.gate_proj                             644      0     1.110e-02    3.312e-03    1.158e-02
  model.layers.46.mlp.up_proj                               644      0     1.150e-02    3.600e-03    1.205e-02
  model.layers.47.mlp.down_proj                             644      0     1.426e-02    2.640e-03    1.450e-02
  model.layers.47.mlp.gate_proj                             644      0     1.055e-02    3.335e-03    1.106e-02
  model.layers.47.mlp.up_proj                               644      0     1.175e-02    3.621e-03    1.229e-02
  model.layers.47.self_attn.k_proj                          644      0     1.241e-02    2.473e-03    1.265e-02
  model.layers.47.self_attn.o_proj                          644      0     1.955e-02    2.555e-03    1.971e-02
  model.layers.47.self_attn.q_proj                          644      0     9.651e-03    2.057e-03    9.869e-03
  model.layers.47.self_attn.v_proj                          644      0     1.057e-02    2.230e-03    1.081e-02
  model.layers.48.linear_attn.in_proj_qkv                   644      0     1.271e-02    2.265e-03    1.291e-02
  model.layers.48.linear_attn.in_proj_z                     644      0     1.209e-02    1.652e-03    1.220e-02
  model.layers.48.linear_attn.out_proj                      644      0     1.715e-02    2.419e-03    1.731e-02
  model.layers.48.mlp.down_proj                             644      0     1.482e-02    2.681e-03    1.506e-02
  model.layers.48.mlp.gate_proj                             644      0     1.034e-02    3.333e-03    1.086e-02
  model.layers.48.mlp.up_proj                               644      0     1.169e-02    3.603e-03    1.223e-02
  model.layers.49.linear_attn.in_proj_qkv                   644      0     1.208e-02    2.277e-03    1.229e-02
  model.layers.49.linear_attn.in_proj_z                     644      0     1.151e-02    1.606e-03    1.162e-02
  model.layers.49.linear_attn.out_proj                      644      0     1.911e-02    2.175e-03    1.923e-02
  model.layers.49.mlp.down_proj                             644      0     1.566e-02    2.696e-03    1.589e-02
  model.layers.49.mlp.gate_proj                             644      0     1.006e-02    3.366e-03    1.061e-02
  model.layers.49.mlp.up_proj                               644      0     1.149e-02    3.562e-03    1.203e-02
  model.layers.5.linear_attn.in_proj_qkv                    644      0     1.246e-02    2.271e-03    1.266e-02
  model.layers.5.linear_attn.in_proj_z                      644      0     1.391e-02    1.910e-03    1.404e-02
  model.layers.5.linear_attn.out_proj                       644      0     1.770e-02    2.208e-03    1.783e-02
  model.layers.5.mlp.down_proj                              644      0     1.529e-02    2.522e-03    1.549e-02
  model.layers.5.mlp.gate_proj                              644      0     1.017e-02    3.533e-03    1.077e-02
  model.layers.5.mlp.up_proj                                644      0     1.128e-02    3.622e-03    1.184e-02
  model.layers.50.linear_attn.in_proj_qkv                   644      0     1.166e-02    2.255e-03    1.188e-02
  model.layers.50.linear_attn.in_proj_z                     644      0     1.085e-02    1.487e-03    1.095e-02
  model.layers.50.linear_attn.out_proj                      644      0     1.816e-02    2.350e-03    1.831e-02
  model.layers.50.mlp.down_proj                             644      0     1.519e-02    2.670e-03    1.542e-02
  model.layers.50.mlp.gate_proj                             644      0     9.845e-03    3.282e-03    1.038e-02
  model.layers.50.mlp.up_proj                               644      0     1.086e-02    3.437e-03    1.140e-02
  model.layers.51.mlp.down_proj                             644      0     1.523e-02    2.655e-03    1.546e-02
  model.layers.51.mlp.gate_proj                             644      0     8.697e-03    3.392e-03    9.334e-03
  model.layers.51.mlp.up_proj                               644      0     1.112e-02    3.553e-03    1.167e-02
  model.layers.51.self_attn.k_proj                          644      0     1.103e-02    2.407e-03    1.130e-02
  model.layers.51.self_attn.o_proj                          644      0     1.923e-02    2.302e-03    1.936e-02
  model.layers.51.self_attn.q_proj                          644      0     9.138e-03    2.022e-03    9.359e-03
  model.layers.51.self_attn.v_proj                          644      0     1.024e-02    2.279e-03    1.049e-02
  model.layers.52.linear_attn.in_proj_qkv                   644      0     9.636e-03    2.113e-03    9.866e-03
  model.layers.52.linear_attn.in_proj_z                     644      0     8.545e-03    1.285e-03    8.642e-03
  model.layers.52.linear_attn.out_proj                      644      0     1.874e-02    2.302e-03    1.888e-02
  model.layers.52.mlp.down_proj                             644      0     1.785e-02    2.686e-03    1.805e-02
  model.layers.52.mlp.gate_proj                             644      0     8.147e-03    3.398e-03    8.827e-03
  model.layers.52.mlp.up_proj                               644      0     1.106e-02    3.565e-03    1.162e-02
  model.layers.53.linear_attn.in_proj_qkv                   644      0     9.508e-03    2.128e-03    9.743e-03
  model.layers.53.linear_attn.in_proj_z                     644      0     8.683e-03    1.312e-03    8.782e-03
  model.layers.53.linear_attn.out_proj                      644      0     1.691e-02    2.275e-03    1.707e-02
  model.layers.53.mlp.down_proj                             644      0     1.847e-02    2.768e-03    1.867e-02
  model.layers.53.mlp.gate_proj                             644      0     7.921e-03    3.402e-03    8.621e-03
  model.layers.53.mlp.up_proj                               644      0     1.097e-02    3.568e-03    1.154e-02
  model.layers.54.linear_attn.in_proj_qkv                   644      0     9.918e-03    2.234e-03    1.017e-02
  model.layers.54.linear_attn.in_proj_z                     644      0     8.291e-03    1.279e-03    8.388e-03
  model.layers.54.linear_attn.out_proj                      644      0     1.747e-02    2.135e-03    1.760e-02
  model.layers.54.mlp.down_proj                             644      0     1.823e-02    2.478e-03    1.839e-02
  model.layers.54.mlp.gate_proj                             644      0     7.279e-03    3.353e-03    8.014e-03
  model.layers.54.mlp.up_proj                               644      0     9.170e-03    3.420e-03    9.788e-03
  model.layers.55.mlp.down_proj                             644      0     1.696e-02    2.628e-03    1.716e-02
  model.layers.55.mlp.gate_proj                             644      0     7.240e-03    3.387e-03    7.994e-03
  model.layers.55.mlp.up_proj                               644      0     1.131e-02    3.603e-03    1.187e-02
  model.layers.55.self_attn.k_proj                          644      0     1.086e-02    2.399e-03    1.112e-02
  model.layers.55.self_attn.o_proj                          644      0     1.727e-02    2.197e-03    1.741e-02
  model.layers.55.self_attn.q_proj                          644      0     8.911e-03    2.105e-03    9.156e-03
  model.layers.55.self_attn.v_proj                          644      0     9.493e-03    2.221e-03    9.748e-03
  model.layers.56.linear_attn.in_proj_qkv                   644      0     9.005e-03    2.094e-03    9.245e-03
  model.layers.56.linear_attn.in_proj_z                     644      0     8.721e-03    1.329e-03    8.821e-03
  model.layers.56.linear_attn.out_proj                      644      0     1.732e-02    2.108e-03    1.745e-02
  model.layers.56.mlp.down_proj                             644      0     1.966e-02    2.529e-03    1.983e-02
  model.layers.56.mlp.gate_proj                             644      0     7.331e-03    3.347e-03    8.060e-03
  model.layers.56.mlp.up_proj                               644      0     1.133e-02    3.598e-03    1.189e-02
  model.layers.57.linear_attn.in_proj_qkv                   644      0     8.637e-03    2.052e-03    8.878e-03
  model.layers.57.linear_attn.in_proj_z                     644      0     8.528e-03    1.300e-03    8.627e-03
  model.layers.57.linear_attn.out_proj                      644      0     1.596e-02    2.503e-03    1.615e-02
  model.layers.57.mlp.down_proj                             644      0     1.632e-02    2.800e-03    1.656e-02
  model.layers.57.mlp.gate_proj                             644      0     7.140e-03    3.324e-03    7.876e-03
  model.layers.57.mlp.up_proj                               644      0     1.115e-02    3.599e-03    1.172e-02
  model.layers.58.linear_attn.in_proj_qkv                   644      0     9.195e-03    2.133e-03    9.439e-03
  model.layers.58.linear_attn.in_proj_z                     644      0     8.769e-03    1.312e-03    8.868e-03
  model.layers.58.linear_attn.out_proj                      644      0     1.887e-02    2.408e-03    1.902e-02
  model.layers.58.mlp.down_proj                             644      0     1.671e-02    2.600e-03    1.692e-02
  model.layers.58.mlp.gate_proj                             644      0     7.023e-03    3.315e-03    7.767e-03
  model.layers.58.mlp.up_proj                               644      0     1.044e-02    3.551e-03    1.103e-02
  model.layers.59.mlp.down_proj                             644      0     1.666e-02    2.500e-03    1.685e-02
  model.layers.59.mlp.gate_proj                             644      0     6.615e-03    3.308e-03    7.396e-03
  model.layers.59.mlp.up_proj                               644      0     9.282e-03    3.449e-03    9.900e-03
  model.layers.59.self_attn.k_proj                          644      0     1.021e-02    2.339e-03    1.048e-02
  model.layers.59.self_attn.o_proj                          644      0     1.797e-02    2.281e-03    1.812e-02
  model.layers.59.self_attn.q_proj                          644      0     8.669e-03    2.119e-03    8.923e-03
  model.layers.59.self_attn.v_proj                          644      0     9.121e-03    2.155e-03    9.374e-03
  model.layers.6.linear_attn.in_proj_qkv                    644      0     1.263e-02    2.268e-03    1.284e-02
  model.layers.6.linear_attn.in_proj_z                      644      0     1.426e-02    1.993e-03    1.440e-02
  model.layers.6.linear_attn.out_proj                       644      0     1.682e-02    2.297e-03    1.698e-02
  model.layers.6.mlp.down_proj                              644      0     1.356e-02    2.399e-03    1.377e-02
  model.layers.6.mlp.gate_proj                              644      0     1.046e-02    3.491e-03    1.102e-02
  model.layers.6.mlp.up_proj                                644      0     1.113e-02    3.563e-03    1.168e-02
  model.layers.60.linear_attn.in_proj_qkv                   644      0     7.945e-03    1.976e-03    8.186e-03
  model.layers.60.linear_attn.in_proj_z                     644      0     7.519e-03    1.195e-03    7.613e-03
  model.layers.60.linear_attn.out_proj                      644      0     1.690e-02    2.503e-03    1.709e-02
  model.layers.60.mlp.down_proj                             644      0     1.588e-02    2.707e-03    1.610e-02
  model.layers.60.mlp.gate_proj                             644      0     6.827e-03    3.241e-03    7.557e-03
  model.layers.60.mlp.up_proj                               644      0     9.185e-03    3.390e-03    9.792e-03
  model.layers.61.linear_attn.in_proj_qkv                   644      0     8.884e-03    2.090e-03    9.126e-03
  model.layers.61.linear_attn.in_proj_z                     644      0     7.958e-03    1.246e-03    8.054e-03
  model.layers.61.linear_attn.out_proj                      644      0     1.672e-02    2.346e-03    1.688e-02
  model.layers.61.mlp.down_proj                             644      0     1.499e-02    2.662e-03    1.522e-02
  model.layers.61.mlp.gate_proj                             644      0     6.562e-03    3.199e-03    7.300e-03
  model.layers.61.mlp.up_proj                               644      0     8.706e-03    3.317e-03    9.314e-03
  model.layers.62.linear_attn.in_proj_qkv                   644      0     8.143e-03    1.972e-03    8.378e-03
  model.layers.62.linear_attn.in_proj_z                     644      0     7.405e-03    1.221e-03    7.506e-03
  model.layers.62.linear_attn.out_proj                      644      0     1.516e-02    2.358e-03    1.534e-02
  model.layers.62.mlp.down_proj                             644      0     1.495e-02    2.536e-03    1.516e-02
  model.layers.62.mlp.gate_proj                             644      0     6.269e-03    3.135e-03    7.008e-03
  model.layers.62.mlp.up_proj                               644      0     6.864e-03    3.147e-03    7.549e-03
  model.layers.63.mlp.down_proj                             644      0     1.190e-02    2.008e-03    1.207e-02
  model.layers.63.mlp.gate_proj                             644      0     5.647e-03    3.070e-03    6.427e-03
  model.layers.63.mlp.up_proj                               644      0     4.713e-03    3.051e-03    5.616e-03
  model.layers.63.self_attn.k_proj                          644      0     1.073e-02    2.414e-03    1.099e-02
  model.layers.63.self_attn.o_proj                          644      0     1.299e-02    2.013e-03    1.315e-02
  model.layers.63.self_attn.q_proj                          644      0     7.009e-03    1.937e-03    7.272e-03
  model.layers.63.self_attn.v_proj                          644      0     4.678e-03    1.730e-03    4.987e-03
  model.layers.7.mlp.down_proj                              644      0     1.419e-02    2.576e-03    1.443e-02
  model.layers.7.mlp.gate_proj                              644      0     1.042e-02    3.567e-03    1.101e-02
  model.layers.7.mlp.up_proj                                644      0     1.136e-02    3.626e-03    1.192e-02
  model.layers.7.self_attn.k_proj                           644      0     1.386e-02    2.708e-03    1.412e-02
  model.layers.7.self_attn.o_proj                           644      0     1.324e-02    2.268e-03    1.343e-02
  model.layers.7.self_attn.q_proj                           644      0     1.249e-02    2.078e-03    1.267e-02
  model.layers.7.self_attn.v_proj                           644      0     1.102e-02    2.326e-03    1.126e-02
  model.layers.8.linear_attn.in_proj_qkv                    644      0     1.245e-02    2.246e-03    1.265e-02
  model.layers.8.linear_attn.in_proj_z                      644      0     1.305e-02    1.811e-03    1.317e-02
  model.layers.8.linear_attn.out_proj                       644      0     1.615e-02    2.282e-03    1.631e-02
  model.layers.8.mlp.down_proj                              644      0     1.507e-02    2.555e-03    1.529e-02
  model.layers.8.mlp.gate_proj                              644      0     1.051e-02    3.565e-03    1.110e-02
  model.layers.8.mlp.up_proj                                644      0     1.130e-02    3.618e-03    1.187e-02
  model.layers.9.linear_attn.in_proj_qkv                    644      0     1.310e-02    2.236e-03    1.329e-02
  model.layers.9.linear_attn.in_proj_z                      644      0     1.360e-02    1.895e-03    1.373e-02
  model.layers.9.linear_attn.out_proj                       644      0     1.599e-02    2.267e-03    1.615e-02
  model.layers.9.mlp.down_proj                              644      0     1.345e-02    2.472e-03    1.368e-02
  model.layers.9.mlp.gate_proj                              644      0     1.059e-02    3.573e-03    1.118e-02
  model.layers.9.mlp.up_proj                                644      0     1.120e-02    3.610e-03    1.177e-02
[i8] per-arm census (int8 kernel calls): {'REF': 0, 'REF2': 0, 'STD': 0, 'INT8': 64000, 'INT8G': 64000, 'INT8G256': 64000, 'INT8GN': 64000, 'SIMROW': 0, 'SIMG128': 0, 'SIMG256': 0, 'SIMG512': 0}
[i8] STD control: fused-reconstruct override consulted 64000 times

ARM RECEIPTS -- what each arm actually executed, from the engine's
own per-call census.  fused/standalone is the reconstruct variant;
acc16/acc32 is the cuBLAS compute type leg B asked hgemm for.
  arm       legB calls          rows     fused  standalone     acc16     acc32   fp32pin
  REF            64000     131072000     64000           0     64000         0         0
  REF2           64000     131072000     64000           0     64000         0         0
  STD            64000     131072000         0       64000     64000         0         0
  INT8           64000     131072000         0       64000     64000         0         0
  INT8G          64000     131072000         0       64000     64000         0         0
  INT8G256       64000     131072000         0       64000     64000         0         0
  INT8GN         64000     131072000         0       64000     64000         0         0
  SIMROW         64000     131072000         0       64000     64000         0         0
  SIMG128        64000     131072000         0       64000     64000         0         0
  SIMG256        64000     131072000         0       64000     64000         0         0
  SIMG512        64000     131072000         0       64000     64000         0         0
  all arms match their expected leg/accumulator signature
  prefill positions aligned: [8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8] per prompt, identical across all 11 arms
[i8] null control: REF2 vs REF max KL is exactly 0

THE TAIL over PREFILL positions only -- one per prefill chunk, so
every arm scores the SAME context. Decode positions are reported
separately and are NOT a numerics measure: once a token flips, the
arms are reading different contexts. The control proves it -- pooled
over decode, STD (which contains no int8 at all) has the same max KL
as INT8, so a pooled tail cannot attribute anything to the kernel.

arm            n        mean         p50         p95         p99         MAX     argmax flip
REF          160   0.000e+00   0.000e+00   0.000e+00   0.000e+00   0.000e+00 0/160 =   0.00%
REF2         160   0.000e+00   0.000e+00   0.000e+00   0.000e+00   0.000e+00 0/160 =   0.00%
STD          160   1.842e-02   2.483e-04   5.761e-03   9.443e-02   2.633e+00 2/160 =   1.25%
INT8         160   5.773e-02   7.039e-04   2.267e-02   2.460e+00   5.107e+00 4/160 =   2.50%
INT8G        160   2.139e-02   4.345e-04   1.508e-02   9.143e-01   1.469e+00 4/160 =   2.50%
INT8G256     160   4.611e-02   5.300e-04   1.404e-02   1.426e+00   3.491e+00 5/160 =   3.12%
INT8GN       160   2.870e-02   9.313e-04   1.932e-02   1.169e+00   1.468e+00 2/160 =   1.25%
SIMROW       160   3.081e-02   6.379e-04   2.664e-02   1.342e+00   2.207e+00 4/160 =   2.50%
SIMG128      160   3.761e-03   3.513e-04   1.497e-02   1.140e-01   1.633e-01 3/160 =   1.88%
SIMG256      160   1.240e-02   3.690e-04   1.796e-02   1.148e-01   1.370e+00 5/160 =   3.12%
SIMG512      160   6.964e-03   2.839e-04   1.333e-02   1.166e-01   6.973e-01 0/160 =   0.00%

Tail attribution -- the 10 worst INT8 prefill positions, with the
control's KL at the SAME position. A tail that is positional shows
the control elevated too; a tail that is the kernel's does not.
   prompt  chunk      INT8 KL       STD KL     SIMROW KL    SIMG128 KL    SIMG256 KL    SIMG512 KL      INT8G KL   INT8G256 KL     INT8GN KL  ratio INT8/STD
        0      1    5.107e+00    2.633e+00     2.207e+00     1.633e-01     7.714e-02     6.973e-01     9.143e-01     3.491e+00     1.079e+00            1.94
       17      2    2.460e+00    9.443e-02     4.935e-01     5.495e-02     6.056e-02     1.166e-01     3.639e-01     9.464e-01     1.468e+00           26.05
        0      7    1.089e+00    6.152e-03     1.342e+00     4.913e-04     1.148e-01     4.359e-02     1.469e+00     1.426e+00     1.169e+00          176.94
       16      6    8.466e-02    3.560e-02     2.070e-02     1.140e-01     1.005e-01     8.082e-03     2.573e-01     1.127e+00     2.384e-01            2.38
       18      7    4.496e-02    7.270e-03     2.254e-02     8.024e-03     1.175e-02     1.333e-02     3.605e-03     5.450e-03     6.169e-03            6.18
       17      5    4.461e-02    2.812e-03     1.859e-02     2.936e-03     1.232e-02     1.786e-02     2.858e-03     4.998e-03     3.867e-02           15.86
        0      6    3.377e-02    1.376e-03     6.161e-02     4.860e-03     1.370e+00     1.779e-02     1.383e-03     2.111e-03     3.812e-02           24.54
        2      7    2.267e-02    1.410e-03     2.664e-02     3.271e-02     3.281e-04     2.323e-03     2.538e-02     8.682e-03     1.932e-02           16.07
        2      6    2.256e-02    2.932e-03     3.521e-03     5.290e-04     3.622e-03     8.354e-03     1.508e-02     1.207e-02     2.661e-01            7.70
       18      6    2.177e-02    4.358e-03     1.803e-03     2.128e-04     9.611e-03     1.151e-03     2.211e-03     1.187e-03     1.099e-03            5.00
  Spearman rank correlation STD vs INT8 across all 160 prefill positions: 0.808

Decode positions (divergence-contaminated, shown for completeness):
arm            n        mean         MAX    greedy agree
REF          160   0.000e+00   0.000e+00 160/160 =  100.0%
REF2         160   0.000e+00   0.000e+00 160/160 =  100.0%
STD          160   7.786e-02   1.225e+01 159/160 =   99.4%
INT8         160   2.455e+00   3.460e+01 138/160 =   86.2%
INT8G        160   1.903e+00   3.252e+01 147/160 =   91.9%
INT8G256     160   1.406e+00   3.238e+01 151/160 =   94.4%
INT8GN       160   2.106e+00   3.276e+01 145/160 =   90.6%
SIMROW       160   1.409e+00   3.248e+01 151/160 =   94.4%
SIMG128      160   4.503e-01   2.116e+01 154/160 =   96.2%
SIMG256      160   4.157e-01   1.660e+01 153/160 =   95.6%
SIMG512      160   4.868e-01   2.628e+01 156/160 =   97.5%

SAMPLER-RELEVANT TAIL @ canonical (T=1.0, top_p=0.95, top_k=20), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF          160     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/160     0.00%
REF2         160     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/160     0.00%
STD          160      98.29%  1.416e-02  4.209e-02  3.583e-01          inf 28/160     4.06%
INT8         160      98.04%  3.231e-02  8.816e-02  8.908e-01          inf 39/160     7.03%
INT8G        160      98.84%  2.481e-02  7.510e-02  8.105e-01          inf 29/160     5.47%
INT8G256     160      98.51%  3.083e-02  6.089e-02  8.545e-01          inf 31/160     6.02%
INT8GN       160      97.61%  2.718e-02  5.838e-02  6.792e-01          inf 39/160     6.02%
SIMROW       160      98.68%  2.887e-02  8.651e-02  7.738e-01          inf 41/160     5.70%
SIMG128      160      98.10%  1.543e-02  5.340e-02  2.277e-01          inf 34/160     3.91%
SIMG256      160      97.95%  2.158e-02  6.428e-02  7.433e-01          inf 36/160     4.77%
SIMG512      160      98.69%  1.533e-02  5.305e-02  3.683e-01          inf 32/160     3.67%

SAMPLER-RELEVANT TAIL @ owner's k~40 (T=1.0, top_p=1.0, top_k=40), prefill positions only.
TV is the headline: under a drafter that tracks the target, speculative
decoding's expected accept rate is sum_t min(p_ref, p_arm), so TV IS the
expected accept-rate loss.
arm            n  setOverlap    TV mean     TV p95     TV max  truncKL p95  bndryChurn  drawDiff
REF          160     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/160     0.00%
REF2         160     100.00%  0.000e+00  0.000e+00  0.000e+00    0.000e+00 0/160     0.00%
STD          160      97.36%  1.396e-02  3.824e-02  3.734e-01          inf 86/160     4.92%
INT8         160      95.68%  3.236e-02  8.102e-02  8.833e-01          inf 122/160     7.34%
INT8G        160      96.67%  2.463e-02  7.549e-02  7.712e-01          inf 110/160     6.80%
INT8G256     160      96.42%  3.034e-02  6.125e-02  8.455e-01          inf 110/160     7.03%
INT8GN       160      95.50%  2.754e-02  6.354e-02  6.749e-01          inf 128/160     8.44%
SIMROW       160      95.50%  2.823e-02  7.762e-02  7.357e-01          inf 118/160     7.66%
SIMG128      160      97.36%  1.457e-02  4.469e-02  2.236e-01          inf 102/160     5.16%
SIMG256      160      96.86%  2.049e-02  6.423e-02  7.454e-01          inf 98/160     6.25%
SIMG512      160      96.98%  1.536e-02  4.803e-02  3.646e-01          inf 106/160     5.62%

FULL-VOCAB numbers below, kept and labelled: they are the right
instrument for GREEDY or beam decode, where every rank matters.

arm         first-prefill-position KL       median          max
REF                         0.000e+00    0.000e+00    0.000e+00
REF2                        0.000e+00    0.000e+00    0.000e+00
STD                         3.772e-04    3.917e-05    3.861e-03
INT8                        5.473e-04    1.630e-04    3.815e-03
INT8G                       3.571e-04    1.079e-04    1.126e-03
INT8G256                    3.289e-04    6.913e-05    1.245e-03
INT8GN                      7.710e-04    1.244e-04    3.963e-03
SIMROW                      3.925e-04    9.799e-05    2.283e-03
SIMG128                     1.693e-03    4.561e-05    2.661e-02
SIMG256                     3.213e-04    5.840e-05    1.627e-03
SIMG512                     1.669e-03    4.478e-05    2.782e-02

PAIRED TV @ canonical (T=1.0, top_p=0.95, top_k=20), prefill positions.
Each row is A scored against B -- B is that ROW's reference, not the
table's.  TV is the expected speculative accept-rate loss, so the
column is read directly in percentage points.  The interval is a
cluster bootstrap over PROMPTS; the IQR is the raw spread over
positions with no floor applied to it.

  A       vs B        n  mean TV pp           95% CI pp    p25 pp    p50 pp    p75 pp    p95 pp    max pp
  REF     REF2      160      0.0000 [  0.0000,  0.0000]    0.0000    0.0000    0.0000    0.0000    0.0000
  REF     STD       160      1.4165 [  0.9713,  1.9410]    0.0000    0.2247    2.0476    4.2088   35.8285
  REF     INT8      160      3.2308 [  1.5625,  5.5451]    0.0000    0.5695    3.2063    8.8158   89.0789
  REF     INT8G     160      2.4809 [  1.2922,  4.0984]    0.0000    0.4951    2.7088    7.5098   81.0508
  REF     INT8G256   160      3.0833 [  1.2332,  5.6830]    0.0000    0.3747    2.4600    6.0893   85.4501
  REF     INT8GN    160      2.7181 [  1.5009,  4.2613]    0.0000    0.7781    3.1552    5.8383   67.9179
  REF     SIMROW    160      2.8873 [  1.3478,  5.3041]    0.0000    0.6594    2.7088    8.6512   77.3848
  REF     SIMG128   160      1.5427 [  1.1176,  2.0207]    0.0000    0.3135    2.4113    5.3403   22.7746
  REF     SIMG256   160      2.1581 [  1.1708,  3.6063]    0.0000    0.5695    2.5569    6.4277   74.3313
  REF     SIMG512   160      1.5326 [  0.9710,  2.2268]    0.0000    0.2958    2.0879    5.3054   36.8292
  REF2    STD       160      1.4165 [  0.9713,  1.9410]    0.0000    0.2247    2.0476    4.2088   35.8285
  REF2    INT8      160      3.2308 [  1.5625,  5.5451]    0.0000    0.5695    3.2063    8.8158   89.0789
  REF2    INT8G     160      2.4809 [  1.2922,  4.0984]    0.0000    0.4951    2.7088    7.5098   81.0508
  REF2    INT8G256   160      3.0833 [  1.2332,  5.6830]    0.0000    0.3747    2.4600    6.0893   85.4501
  REF2    INT8GN    160      2.7181 [  1.5009,  4.2613]    0.0000    0.7781    3.1552    5.8383   67.9179
  REF2    SIMROW    160      2.8873 [  1.3478,  5.3041]    0.0000    0.6594    2.7088    8.6512   77.3848
  REF2    SIMG128   160      1.5427 [  1.1176,  2.0207]    0.0000    0.3135    2.4113    5.3403   22.7746
  REF2    SIMG256   160      2.1581 [  1.1708,  3.6063]    0.0000    0.5695    2.5569    6.4277   74.3313
  REF2    SIMG512   160      1.5326 [  0.9710,  2.2268]    0.0000    0.2958    2.0879    5.3054   36.8292
  STD     INT8      160      3.2851 [  1.6166,  5.5859]    0.0000    0.5375    3.0925    9.7327   89.0789
  STD     INT8G     160      2.6943 [  1.2979,  4.6769]    0.0000    0.3439    2.3805    6.7241   81.0508
  STD     INT8G256   160      2.8819 [  1.1536,  5.5011]    0.0000    0.5460    2.4506    6.0087   82.9677
  STD     INT8GN    160      2.7129 [  1.5138,  4.1883]    0.0000    0.6738    3.1113    7.5858   71.9267
  STD     SIMROW    160      2.8497 [  1.3452,  5.3837]    0.0000    0.6154    2.8304    5.6719   80.4626
  STD     SIMG128   160      1.3488 [  0.9818,  1.7583]    0.0000    0.3912    2.2644    4.2088   17.3232
  STD     SIMG256   160      2.3982 [  1.1788,  4.2033]    0.0000    0.3273    2.6156    6.2382   74.3313
  STD     SIMG512   160      1.2376 [  0.8689,  1.6451]    0.0000    0.2034    1.7893    4.8737   17.6642
  INT8    INT8G     160      2.8239 [  1.4329,  4.6055]    0.0000    0.5227    2.7070    8.3326   89.0789
  INT8    INT8G256   160      3.1718 [  1.4585,  5.2321]    0.0000    0.5891    2.6652    8.3326   97.2010
  INT8    INT8GN    160      3.0962 [  1.6373,  5.0145]    0.0000    0.5460    3.3230    7.5858   89.0789
  INT8    SIMROW    160      3.4039 [  1.6404,  5.7997]    0.0000    0.9739    3.0983    8.5535   92.2928
  INT8    SIMG128   160      3.2921 [  1.6314,  5.6549]    0.0000    0.6824    3.3894    7.8393   89.0789
  INT8    SIMG256   160      3.8352 [  1.8209,  6.8615]    0.0000    0.9011    3.4417    9.8541   89.0789
  INT8    SIMG512   160      3.2169 [  1.6516,  5.3677]    0.0000    0.8311    3.0925    7.9417   89.5440
  INT8G   INT8G256   160      2.5266 [  1.1809,  4.1349]    0.0000    0.1450    2.1208    8.3009   70.2543
  INT8G   INT8GN    160      2.6058 [  1.3356,  4.1326]    0.0000    0.4640    2.7875    7.9358   72.7620
  INT8G   SIMROW    160      2.8572 [  1.6293,  4.3072]    0.0000    0.5705    3.1123   12.0336   61.9965
  INT8G   SIMG128   160      2.7162 [  1.4096,  4.5924]    0.0000    0.2279    2.8511    7.1137   81.0508
  INT8G   SIMG256   160      3.0303 [  1.4629,  5.4211]    0.0000    0.8180    2.8896    9.2085   74.3313
  INT8G   SIMG512   160      2.7377 [  1.3557,  4.6339]    0.0000    0.3520    2.8821    6.7052   70.3817
  INT8G256 INT8GN    160      3.3266 [  1.5481,  5.4033]    0.0000    0.3747    2.7594    8.9386   94.8272
  INT8G256 SIMROW    160      2.5214 [  1.4141,  3.9090]    0.0000    0.3905    3.2015    6.6532   69.9098
  INT8G256 SIMG128   160      3.0059 [  1.3223,  5.5747]    0.0000    0.5636    2.8962    6.0087   86.9569
  INT8G256 SIMG256   160      3.8216 [  1.5124,  7.2637]    0.0000    0.5205    2.8844    8.7525   79.9929
  INT8G256 SIMG512   160      3.0100 [  1.2528,  5.6255]    0.0000    0.5695    2.4489    6.7547   89.2858
  INT8GN  SIMROW    160      3.1729 [  1.6034,  5.1158]    0.0000    0.8311    3.1492    9.4669   89.9190
  INT8GN  SIMG128   160      2.8552 [  1.6209,  4.3602]    0.0000    0.8033    3.1897    7.8439   70.3544
  INT8GN  SIMG256   160      3.2556 [  1.6387,  5.6586]    0.0000    0.6129    3.3391    9.4693   74.3313
  INT8GN  SIMG512   160      2.6612 [  1.4489,  4.1746]    0.0000    0.6665    2.7517    6.9855   79.1143
  SIMROW  SIMG128   160      3.0081 [  1.4532,  5.5070]    0.0000    0.9780    2.9278    7.7539   77.3848
  SIMROW  SIMG256   160      3.6026 [  1.6071,  6.8477]    0.0000    0.8334    3.0001    9.3907   74.3313
  SIMROW  SIMG512   160      2.7702 [  1.3316,  5.2233]    0.0000    0.8311    2.8663    6.0165   83.7390
  SIMG128 SIMG256   160      2.2980 [  1.1854,  3.9779]    0.0000    0.6290    2.4741    6.7886   74.3313
  SIMG128 SIMG512   160      1.5412 [  1.0574,  2.0839]    0.0000    0.6042    2.6274    5.6719   17.3866
  SIMG256 SIMG512   160      2.4273 [  1.2657,  4.1254]    0.0000    0.3018    2.7427    6.7547   74.3313

  TV distance matrix, mean TV pp (symmetric):
          REF      REF2       STD      INT8     INT8G  INT8G256    INT8GN    SIMROW   SIMG128   SIMG256   SIMG512
  REF        0.0000    0.0000    1.4165    3.2308    2.4809    3.0833    2.7181    2.8873    1.5427    2.1581    1.5326
  REF2       0.0000    0.0000    1.4165    3.2308    2.4809    3.0833    2.7181    2.8873    1.5427    2.1581    1.5326
  STD        1.4165    1.4165    0.0000    3.2851    2.6943    2.8819    2.7129    2.8497    1.3488    2.3982    1.2376
  INT8       3.2308    3.2308    3.2851    0.0000    2.8239    3.1718    3.0962    3.4039    3.2921    3.8352    3.2169
  INT8G      2.4809    2.4809    2.6943    2.8239    0.0000    2.5266    2.6058    2.8572    2.7162    3.0303    2.7377
  INT8G256    3.0833    3.0833    2.8819    3.1718    2.5266    0.0000    3.3266    2.5214    3.0059    3.8216    3.0100
  INT8GN     2.7181    2.7181    2.7129    3.0962    2.6058    3.3266    0.0000    3.1729    2.8552    3.2556    2.6612
  SIMROW     2.8873    2.8873    2.8497    3.4039    2.8572    2.5214    3.1729    0.0000    3.0081    3.6026    2.7702
  SIMG128    1.5427    1.5427    1.3488    3.2921    2.7162    3.0059    2.8552    3.0081    0.0000    2.2980    1.5412
  SIMG256    2.1581    2.1581    2.3982    3.8352    3.0303    3.8216    3.2556    3.6026    2.2980    0.0000    2.4273
  SIMG512    1.5326    1.5326    1.2376    3.2169    2.7377    3.0100    2.6612    2.7702    1.5412    2.4273    0.0000

  TV distance matrix, median TV pp (symmetric):
          REF      REF2       STD      INT8     INT8G  INT8G256    INT8GN    SIMROW   SIMG128   SIMG256   SIMG512
  REF        0.0000    0.0000    0.2247    0.5695    0.4951    0.3747    0.7781    0.6594    0.3135    0.5695    0.2958
  REF2       0.0000    0.0000    0.2247    0.5695    0.4951    0.3747    0.7781    0.6594    0.3135    0.5695    0.2958
  STD        0.2247    0.2247    0.0000    0.5375    0.3439    0.5460    0.6738    0.6154    0.3912    0.3273    0.2034
  INT8       0.5695    0.5695    0.5375    0.0000    0.5227    0.5891    0.5460    0.9739    0.6824    0.9011    0.8311
  INT8G      0.4951    0.4951    0.3439    0.5227    0.0000    0.1450    0.4640    0.5705    0.2279    0.8180    0.3520
  INT8G256    0.3747    0.3747    0.5460    0.5891    0.1450    0.0000    0.3747    0.3905    0.5636    0.5205    0.5695
  INT8GN     0.7781    0.7781    0.6738    0.5460    0.4640    0.3747    0.0000    0.8311    0.8033    0.6129    0.6665
  SIMROW     0.6594    0.6594    0.6154    0.9739    0.5705    0.3905    0.8311    0.0000    0.9780    0.8334    0.8311
  SIMG128    0.3135    0.3135    0.3912    0.6824    0.2279    0.5636    0.8033    0.9780    0.0000    0.6290    0.6042
  SIMG256    0.5695    0.5695    0.3273    0.9011    0.8180    0.5205    0.6129    0.8334    0.6290    0.0000    0.3018
  SIMG512    0.2958    0.2958    0.2034    0.8311    0.3520    0.5695    0.6665    0.8311    0.6042    0.3018    0.0000

lm_head CUT -- the divergence entering the head vs leaving it,
prefill positions, relative rms.  'in' is the residual stream after
24 layers; 'out' is the centred logits.  amp = out/in: at 1.0 the
head passes the body's divergence through unchanged, above it the
head adds its own.

  A       vs B        n   rel rms IN  rel rms OUT      amp
  REF     REF2      160   0.0000e+00   0.0000e+00      nan
  REF     STD       160   3.5182e-02   3.2698e-02    0.929
  REF     INT8      160   6.8381e-02   6.2660e-02    0.916
  REF     INT8G     160   5.3433e-02   4.9346e-02    0.923
  REF     INT8G256   160   5.5208e-02   5.0631e-02    0.917
  REF     INT8GN    160   6.8677e-02   6.2673e-02    0.913
  REF     SIMROW    160   6.8020e-02   6.2393e-02    0.917
  REF     SIMG128   160   3.8032e-02   3.5635e-02    0.937
  REF     SIMG256   160   4.3483e-02   3.9778e-02    0.915
  REF     SIMG512   160   3.9878e-02   3.6752e-02    0.922
  REF2    STD       160   3.5182e-02   3.2698e-02    0.929
  REF2    INT8      160   6.8381e-02   6.2660e-02    0.916
  REF2    INT8G     160   5.3433e-02   4.9346e-02    0.923
  REF2    INT8G256   160   5.5208e-02   5.0631e-02    0.917
  REF2    INT8GN    160   6.8677e-02   6.2673e-02    0.913
  REF2    SIMROW    160   6.8020e-02   6.2393e-02    0.917
  REF2    SIMG128   160   3.8032e-02   3.5635e-02    0.937
  REF2    SIMG256   160   4.3483e-02   3.9778e-02    0.915
  REF2    SIMG512   160   3.9878e-02   3.6752e-02    0.922
  STD     INT8      160   7.4994e-02   6.9261e-02    0.924
  STD     INT8G     160   5.7934e-02   5.3724e-02    0.927
  STD     INT8G256   160   5.8550e-02   5.3825e-02    0.919
  STD     INT8GN    160   7.5740e-02   6.9460e-02    0.917
  STD     SIMROW    160   7.4294e-02   6.8322e-02    0.920
  STD     SIMG128   160   4.6438e-02   4.3886e-02    0.945
  STD     SIMG256   160   4.6721e-02   4.3031e-02    0.921
  STD     SIMG512   160   4.4921e-02   4.1983e-02    0.935
  INT8    INT8G     160   6.2359e-02   5.7061e-02    0.915
  INT8    INT8G256   160   6.6358e-02   6.0205e-02    0.907
  INT8    INT8GN    160   6.8872e-02   6.2358e-02    0.905
  INT8    SIMROW    160   7.9982e-02   7.2488e-02    0.906
  INT8    SIMG128   160   7.0351e-02   6.4092e-02    0.911
  INT8    SIMG256   160   7.2744e-02   6.5778e-02    0.904
  INT8    SIMG512   160   7.0410e-02   6.4170e-02    0.911
  INT8G   INT8G256   160   4.7056e-02   4.3812e-02    0.931
  INT8G   INT8GN    160   6.3727e-02   5.8584e-02    0.919
  INT8G   SIMROW    160   7.2930e-02   6.6006e-02    0.905
  INT8G   SIMG128   160   5.8858e-02   5.4527e-02    0.926
  INT8G   SIMG256   160   6.3751e-02   5.8866e-02    0.923
  INT8G   SIMG512   160   6.4821e-02   5.9794e-02    0.922
  INT8G256 INT8GN    160   6.6562e-02   6.0787e-02    0.913
  INT8G256 SIMROW    160   7.1880e-02   6.5436e-02    0.910
  INT8G256 SIMG128   160   6.3678e-02   5.8870e-02    0.924
  INT8G256 SIMG256   160   6.2935e-02   5.7974e-02    0.921
  INT8G256 SIMG512   160   6.6380e-02   6.0796e-02    0.916
  INT8GN  SIMROW    160   7.9253e-02   7.1486e-02    0.902
  INT8GN  SIMG128   160   7.0630e-02   6.4754e-02    0.917
  INT8GN  SIMG256   160   7.1101e-02   6.4509e-02    0.907
  INT8GN  SIMG512   160   6.8460e-02   6.2106e-02    0.907
  SIMROW  SIMG128   160   6.7945e-02   6.2127e-02    0.914
  SIMROW  SIMG256   160   6.9007e-02   6.2559e-02    0.907
  SIMROW  SIMG512   160   7.0774e-02   6.4673e-02    0.914
  SIMG128 SIMG256   160   4.9244e-02   4.4915e-02    0.912
  SIMG128 SIMG512   160   4.4175e-02   4.0503e-02    0.917
  SIMG256 SIMG512   160   4.4342e-02   4.0156e-02    0.906

PER-TOKEN AMAX -- is the crest factor of the quantised row what
selects the tail?  crest = amax/rms of the int8 GEMM's input row;
under the uniform-residual model the relative noise the row picks
up is crest/(127*sqrt(12)) = crest/440, and qerr is that noise
measured directly by round-tripping the scored row through int8.
arm supplying the statistics: INT8

  statistic                                  p50         p95         p99         max
  crest, scored row (max/linear)       1.174e+01   1.404e+01   1.682e+01   1.761e+01
  crest, any row in chunk              1.891e+01   2.119e+01   2.319e+01   2.337e+01
  qerr, scored row (rel rms)           2.662e-02   3.197e-02   3.801e-02   3.959e-02
  n positions: 160

What the per-token scale is paying for, over the whole census.
crest_g is the crest a per-128-group scale would leave --
sqrt(mean_b amax_b^2)/rms -- and its ratio to crest is the noise
reduction available.  conc is max/median of the per-128-block rms:
at 1 the row is flat and grouping buys nothing.
  statistic                                  p50         p95         p99         max
  crest_g (per-128 scale)                  2.603       2.878       2.934       3.162
  crest / crest_g (noise cut)              2.161       4.127       5.381       9.408
  block rms concentration                  3.314      12.029      24.162      76.885
  n (class, position) samples: 64000
  heaviest-block STABILITY per class (share of positions whose argmax block is that class's modal block):
    p05=0.037 p50=0.544 p95=1.000 over 400 classes
    a static outlier split needs this near 1.0; at 1/nblocks (~0.025 for K=5120) the heavy block is a different block every token and no static partition exists

The 10 worst INT8 positions, with the crest factor at each.  If the
per-token amax is the mechanism, these are the extreme-crest rows.
   prompt  chunk      INT8 KL   crest row  crest chunk        qerr  worst linear
        0      1    5.107e+00      11.213       18.112   2.543e-02  L8.linear_attn.out_proj
       17      2    2.460e+00      11.934       19.707   2.733e-02  L50.linear_attn.out_proj
        0      7    1.089e+00      16.305       20.238   3.729e-02  L56.linear_attn.out_proj
       16      6    8.466e-02      11.372       19.141   2.586e-02  L21.linear_attn.out_proj
       18      7    4.496e-02       9.961       18.478   2.269e-02  L17.linear_attn.out_proj
       17      5    4.461e-02      10.826       19.799   2.454e-02  L51.self_attn.o_proj
        0      6    3.377e-02      10.212       19.009   2.322e-02  L22.linear_attn.out_proj
        2      7    2.267e-02      10.866       21.451   2.455e-02  L58.linear_attn.out_proj
        2      6    2.256e-02      12.691       21.755   2.882e-02  L2.linear_attn.out_proj
       18      6    2.177e-02      17.611       19.024   3.959e-02  L58.linear_attn.out_proj
  ... and the 10 MEDIAN INT8 positions, as the contrast:
        0      5    5.335e-04      13.025       18.653   2.975e-02  L2.linear_attn.out_proj
        1      2    5.376e-04      11.424       18.076   2.582e-02  L0.mlp.down_proj
       18      0    5.999e-04      12.357       18.953   2.801e-02  L3.mlp.down_proj
       14      7    6.118e-04      11.801       18.312   2.694e-02  L38.linear_attn.out_proj
       18      1    6.745e-04      11.799       20.317   2.662e-02  L0.mlp.down_proj
        6      0    7.039e-04      11.325       18.915   2.571e-02  L31.self_attn.o_proj
        7      7    7.503e-04      10.683       17.944   2.443e-02  L2.mlp.down_proj
        4      1    7.993e-04      12.934       19.746   2.866e-02  L61.linear_attn.out_proj
        8      5    8.050e-04       9.350       16.968   2.125e-02  L61.linear_attn.out_proj
        1      4    8.112e-04      12.907       18.914   2.894e-02  L58.linear_attn.out_proj

Is the tail MORE NOISE or MORE RESPONSE?  rel rms of the residual
stream entering lm_head, per position, and the INT8/STD ratio.
Flat ratio across the tail => the position amplifies every arm and
int8 is merely 2.4x noisier everywhere.  Ratio spiking on the tail
=> int8 injects locally more noise there, which is a mechanism.
  ratio over ALL 160 positions: p50=2.077 p95=3.187 max=8.235
  ratio at the 10 worst INT8 positions: min=1.597 median=2.517 max=8.235
   prompt  chunk      INT8 KL  relIN INT8   relIN STD    ratio
        0      1    5.107e+00  1.1541e+00  6.8820e-01    1.677
       17      2    2.460e+00  6.4971e-01  7.8898e-02    8.235
        0      7    1.089e+00  5.4671e-01  1.6495e-01    3.314
       16      6    8.466e-02  1.4382e-01  9.0069e-02    1.597
       18      7    4.496e-02  6.6619e-02  2.1530e-02    3.094
       17      5    4.461e-02  8.6996e-02  3.1929e-02    2.725
        0      6    3.377e-02  1.1258e+00  3.9821e-01    2.827
        2      7    2.267e-02  4.1664e-01  2.5931e-01    1.607
        2      6    2.256e-02  7.5557e-02  4.1868e-02    1.805
       18      6    2.177e-02  6.6717e-02  2.8902e-02    2.308
  Spearman(relIN ratio, INT8 KL) = 0.120   Spearman(relIN INT8, INT8 KL) = 0.164

Rank correlation of the crest factor with each arm's KL.  STD is the
NULL CONTROL: it runs no quantiser, so the crest factor cannot cause
its KL.  A crest that ranks STD as well as it ranks INT8 is ranking
hard POSITIONS, not the quantiser, and explains nothing.
  crest statistic                    REF       REF2        STD       INT8      INT8G   INT8G256     INT8GN     SIMROW    SIMG128    SIMG256    SIMG512  INT8 excess
  crest, scored row               -0.016     -0.016     -0.029      0.016     -0.042      0.017      0.033     -0.008      0.030      0.069     -0.004        0.089
  crest, any row                  -0.155     -0.155      0.110      0.200      0.159      0.147      0.161      0.166      0.163      0.148      0.181        0.181
  qerr, scored row                -0.002     -0.002     -0.032      0.015     -0.045      0.011      0.034     -0.009      0.031      0.069     -0.002        0.091

Mean crest of the scored row, by linear class -- the 15 heaviest.
A class well above the Gaussian-rotation expectation (~4.5 for a
128-wide Hadamard over K=5120) is a class the per-token scale is
wasting levels on.
  linear class                                        mean crest         p95         max
  L2.mlp.down_proj                                         8.770      11.690      13.575
  L2.linear_attn.out_proj                                  8.289      11.635      13.623
  L0.mlp.down_proj                                         8.150      11.636      12.454
  L58.linear_attn.out_proj                                 8.059      13.775      17.611
  L4.linear_attn.out_proj                                  7.805      11.029      13.589
  L5.linear_attn.out_proj                                  7.679      11.765      14.896
  L1.linear_attn.out_proj                                  7.297      10.829      13.089
  L17.linear_attn.out_proj                                 7.105       9.961      13.289
  L56.linear_attn.out_proj                                 7.104      11.531      16.305
  L49.linear_attn.in_proj_z                                7.064       7.925       8.241
  L44.linear_attn.in_proj_z                                7.024       7.967       8.572
  L61.linear_attn.out_proj                                 7.000      10.387      12.934
  L6.linear_attn.out_proj                                  6.997       9.994      11.514
  L45.linear_attn.in_proj_z                                6.995       7.790       8.149
  L46.linear_attn.in_proj_z                                6.979       7.770       8.086

saved per-arm prefill logits+hidden to /work/tools/int8_gemm/logits_pergroup.pt
