# Why gate3_kl cannot score this checkpoint, and which path is responsible (#1861).
#   RTX 4090 (sm_89), registry.arbi.work/arbi-serve:test-latest.
#   This is a specific, module-level attribution for the OPEN issue #1426,
#   "Greedy decoding is not reproducible", which is on this same checkpoint.
#
# gate3_kl refuses to attribute any arm's KL unless REF2 scores EXACTLY 0
# against REF. On this checkpoint it scores 1.028e-02, so the harness stops --
# correctly. The question that matters is WHICH module broke the control.
#
# $ python3 gate3_kl.py --model <...> --bits 6 --shape 27 \
#       --arms REF,REF2,STD,INT8 --n-prompts 10 --target-tokens 4096
#   [i8] kernel census: int8 calls=2320 rows=4751360 reconstructs SKIPPED=2320
#                       fallback (not K=6) calls=0
#   [i8] shapes served by the kernel (K,N)->calls:
#        {(2688,256): 240, (2688,3712): 460, (2688,4096): 120,
#         (2688,10368): 460, (3712,2688): 460, (4096,2688): 580}
#   [i8] per-arm census: {'REF': 0, 'REF2': 0, 'STD': 0, 'INT8': 2320}
#   RuntimeError: the REF2 null control scored max KL 1.028e-02 against REF
#
# THE ARM DID RUN. 2320 int8 calls, zero fallbacks, and the six geometries
# served are EXACTLY the six the census rules servable -- measured from the
# engine rather than from the safetensors. So the refusal is not about reach;
# it is about the reference.
#
# READ THE FIRST DIVERGENCE, NEVER THE SET. The set is not stable and must not
# be reported as one. A divergence born in an early layer is usually ABSORBED
# by the next rounding to fp16/bf16 and leaves no trace; occasionally it
# survives, and from there it propagates into every module for the rest of the
# network. Below, the same model on the same input gives 3 differing modules on
# one pair and 319 on another, with no change in the cause. An earlier version
# of this receipt reported ONE pair -- it happened to be a 4-module one -- and
# concluded "all 93 dense projections are byte-identical". That conclusion was
# an artefact of where the divergence happened to die on that pair, and it is
# withdrawn: on a propagating pair the dense projections differ too, because
# their INPUT differs. Counting byte-identical modules downstream of an
# absorbed divergence is evidence of luck, not of a clean path.
#
# WHAT SURVIVES REPETITION is the first divergence in execution order, and it
# is the same kind every time: a ROUTED expert block, 6 of 6 pairs below and
# 6 of 6 in an independent earlier run of the same tool. Every module that ran
# before it is byte-identical.
#
# THAT IS A NEGATIVE FOR BOTH OF #1426's REMAINING CANDIDATES. That issue had
# narrowed the cause to the GDN/Mamba chunked scan or the multi-query prefill
# attention kernel. Layer 0 is a mamba block and it is upstream of the first
# divergence in EVERY pair recorded here, so the chunked scan ran reproducibly
# every time. The strongest single observation is from the earlier run, whose
# deepest pair did not diverge until layer 43: 267 modules byte-identical
# first, covering five attention blocks and nineteen mamba blocks. Neither
# candidate can be the source of a divergence that appears after both have run
# clean.
#
# The int8 prefill leg does not touch the routed path: routed experts are out
# of scope for #1861 and are served by exl3_moe_fused, not by this kernel.

# /mnt/k8scache/models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-exl3-4bpw
# 1024-token prefill, 6 pairs, compared by byte hash

pair 0: 331 modules compared, 264 differ; first in execution order = backbone.layers.1.mixer.experts (9 modules upstream of it, all byte-identical)
pair 1: 331 modules compared, 3 differ; first in execution order = backbone.layers.1.mixer.experts (9 modules upstream of it, all byte-identical)
pair 2: 331 modules compared, 4 differ; first in execution order = backbone.layers.6.mixer.experts (38 modules upstream of it, all byte-identical)
pair 3: 331 modules compared, 4 differ; first in execution order = backbone.layers.1.mixer.experts (9 modules upstream of it, all byte-identical)
pair 4: 331 modules compared, 263 differ; first in execution order = backbone.layers.6.mixer.experts (38 modules upstream of it, all byte-identical)
pair 5: 331 modules compared, 264 differ; first in execution order = backbone.layers.1.mixer.experts (9 modules upstream of it, all byte-identical)

first divergence, by kind, across pairs:
  6/6  routed experts

NOT REPRODUCIBLE
