# The routed MoE combine: mechanism confirmed, and what determinism costs (#1922).
#   NVIDIA-Nemotron-3.5-Lightning-30B-A3B-exl3-4bpw, RTX 4090 (sm_89),
#   registry.arbi.work/arbi-serve:test-latest, sole tenant.
#
# THE MECHANISM, CONFIRMED BEFORE ANYTHING WAS BUILT ON IT.
# Every (expert, token) contribution is added into a zeroed fp32 output by
# the four atomicAdds ending had_hf_r_128_d_inner (exllamav3
# hadamard_inner.cuh:418-473), called from had_d_out in exl3_moe_kernel.cuh.
# The order they arrive in is set by a dynamic ticket queue
#   sched[2 + group_idx] = num_groups + atomicAdd(&sched[0], 1)
# (exl3_moe_kernel.cuh:266), so it varies per launch. fp32 addition is not
# associative. The untraced epilogue named when #1922 was filed IS this
# one -- it is a single path, not a second source.
#
# TWO ARMS THAT COULD EACH HAVE KILLED THAT STORY, AND DID NOT:
#   TOPK1  one expert per token, so no two contributions share an output
#          element and no order exists to vary.
#   CONC1  one concurrent group, so the ticket queue has a single drawer
#          and hands experts out in a fixed order.
# BASE is the control in the other direction: it MUST diverge, or the probe
# is not reproducing the defect and a clean arm means nothing.

$ python3 tools/moe_determinism/nondet_probe.py --arms BASE,TOPK1,CONC1 --repeats 4
  BASE   4/4 pairs diverged   first divergence a routed mixer.experts every time
  TOPK1  0/4 pairs diverged   byte-identical
  CONC1  0/4 pairs diverged   byte-identical

# THE DEFECT IS ONE KERNEL'S, NOT THE MoE's. This repo ships three routed
# legs and only the expert-major one (exl3_moe) is affected:

$ python3 tools/moe_determinism/nondet_probe.py --arms BASE,GROUPED,PEREXPERT --repeats 3
  BASE       3/3 pairs diverged
  GROUPED    0/3 pairs diverged   byte-identical  (_forward_grouped)
  PEREXPERT  0/3 pairs diverged   byte-identical  (per-expert modules)

# WHICH ALSO EXPLAINS #1426's OWN FINDING. _use_expert_major selects the
# affected leg ABOVE moe_expert_major_min_rows rows, so decode takes a
# deterministic leg and multi-token prefill takes the broken one. That is
# exactly the localisation #1426 measured from the outside, and why
# --chunk-prefill=1 was reproducible.

# ===================================================================
# WHAT DETERMINISM COSTS. This is the part that decides the question.
# Routed dispatch in isolation, one layer, interleaved round-robin,
# 40 reps (tools/moe_determinism/bench_combine.py):

  rows arm             median ms      IQR  vs expert-major   determinism
------------------------------------------------------------------------
  2048 EXPERT_MAJOR       6.1778   0.0280                             NO
  2048 CONC1             22.8639   0.1075         +270.10%           yes
  2048 GROUPED           65.1311   0.0666         +954.28%           yes
  2048 PEREXPERT         46.8669   4.5180         +658.64%           yes


# AND THE NUMBER THE DECISION ACTUALLY RESTS ON -- whole prefill, 2048
# tokens, one token out, so no generated content enters the timing
# (tools/moe_determinism/bench_prefill.py, 12 interleaved reps):

arm             median ms      IQR  vs expert-major  determinism
----------------------------------------------------------------
EXPERT_MAJOR       293.85     0.38                            NO
CONC1              592.04     0.22         +101.48%          yes
GROUPED           1562.26     2.10         +431.66%          yes

# THE CHEAPEST DETERMINISTIC OPTION AVAILABLE TODAY DOUBLES PREFILL, so
# there is no always-on fix here to ship. What ships instead is the mode:
# ARBI_MOE_DETERMINISTIC_COMBINE / `moe_deterministic_combine`, default OFF,
# pinning the kernel's group concurrency to 1. It is for KL/TV harnesses and
# reproducibility work -- gate3_kl cannot attribute ANY arm on a checkpoint
# whose null control fails, so without this there is no quality gate on a MoE
# checkpoint at all -- and it is not a serving default.
#
# ===================================================================
# THE SHIPPED FLAG, EXERCISED THE WAY A DEPLOYMENT SETS IT.
# Not the monkeypatch that stood in for it while it did not exist: the FLAG
# arm sets the runtime flag and lets the dispatch read it.
#
# $ python3 tools/moe_determinism/nondet_probe.py --arms BASE,FLAG --repeats 5
#   BASE   pair 0: 331 compared,   5 differ; first = backbone.layers.1.mixer.experts
#   BASE   pair 1: 331 compared,   3 differ; first = backbone.layers.3.mixer.experts
#   BASE   pair 2: 331 compared, 319 differ; first = backbone.layers.1.mixer.experts
#   BASE   pair 3: 331 compared,   5 differ; first = backbone.layers.1.mixer.experts
#   BASE   pair 4: 331 compared, 319 differ; first = backbone.layers.1.mixer.experts
#   FLAG   pair 0..4: 331 compared, 0 differ; byte-identical
#   counter moe_deterministic_combine_read fired: 138
#   BASE 5/5   FLAG 0/5
#
# THREE THINGS IN THAT OUTPUT ARE CONTROLS, NOT DECORATION:
#  * BASE MUST diverge. It is 5/5. A probe that cannot reproduce the defect
#    cannot report a fix, and the harness refuses if BASE comes back clean.
#  * THE COMPARED COUNT must match the control's. An earlier revision of the
#    FLAG arm raised partway through the forward and compared 9 modules
#    instead of 331 -- and "0 differ" printed exactly like a clean sheet. The
#    probe now refuses an arm that covered materially less than the control.
#  * THE COUNTER must be non-zero. 138 = 23 MoE layers x 6 forwards, so the
#    dispatch really took the pinned-concurrency branch. The flag being SET is
#    not evidence that anything read it, which is the whole reason the counter
#    exists.
#
# What five pairs bound: five pairs. This is not a determinism proof, it is
# the defect no longer reproducing where it reproduced 5/5 immediately before,
# under a lever whose engagement is separately counted.
