iter 13 — permutation fixes the max, but calibrates the wrong null QUANTIFIED

Attack item R-7 · 2026-07-30 · harness harness/permutation_reference_for_max.py

What was recommended, and why it is the right shape

The research's decision (b)/(c) recommends replacing the fixed 0.50 line on xi_worst with a permutation reference for the same max the rule uses, with Lin–Han's (1+#)/(1+B) smoothing. Chatterjee has no closed form even for a two-element max and says a permutation test is easy. Permuting the candidate within each cell absorbs, without modelling: the max-of-N selection bias (iter 9), the cell dependence that damps it (iter 9), and the per-cell heterogeneity in n, ties and attainable ceiling (iters 2, 10).

Prediction, written before running

Size-valid by construction; power enormous; and therefore answering the wrong question — because shuffling destroys all association, so the reference is "ξ = 0", while the cascade's rule is "ξ_worst < 0.50", a practical-redundancy threshold. The substantive half held. The size half did not.

Result

  dep_c  reps  mean xi/cell  mean xi_worst  perm rejects  fixed-line bans
   0.00   120       -0.0004         0.0651           9%               0%
   0.02    40        0.0007         0.0662           2%               0%
   0.05    40        0.0019         0.0656           5%               0%
   0.10    40        0.0048         0.0718          12%               0%
   0.20    40        0.0209         0.0877          25%               0%

rule	reaches 90% power at
permutation reference	per-cell ξ ≈ 0.094
fixed 0.50 line	never fires — 0% across the entire sweep

Iteration 9 measured that the fixed line needs per-cell ξ ≈ 0.45 before the max clears 0.50. So the two rules fire roughly 5× apart, and the substantive claim stands:

The max-selection problem and the threshold problem are separate. A permutation reference solves the first — it calibrates the max correctly, absorbing selection bias, dependence and heterogeneity with no sd estimate and no Gaussian assumption. It cannot solve the second, because it calibrates "ξ = 0" and the cascade needs "ξ < 0.50". Adopting it as the redundancy gate would reject nearly every real candidate: real features are never exactly independent.

My own size check was too lenient, and the size is inflated

The harness printed VALID. It should not have. Read strictly:

observed 0.092   nominal 0.050   se 0.0199
deviation = 2.09 se   -> passes at 3se, FAILS at 2se
exact binomial P(>= 11 rejections | p=0.05, n=120) = 0.0384

A permutation test is supposed to be exact. An observed size of 9.2% against a nominal 5%, with an exact-binomial tail probability of 0.038, is not a clean pass — and my script's 3σ criterion was loose enough to wave it through. Recorded here rather than quietly re-tuned.

The unresolved design question behind it

I permute each cell independently. The observed cells are overlapping windows of one master series, so their ξ values are correlated; the permuted cells are not. That mismatch means the reference distribution is built under a different dependence structure than the observed statistic.

The naive expectation is that this would make the test conservative (independent cells → higher max → observed less likely to exceed → fewer rejections). The measurement went the other way. I do not have an explanation, and I am not going to invent one. What it establishes is that a permutation reference for a max over dependent cells requires a design decision the research's recommendation does not address: what, exactly, gets permuted. Permuting cells jointly to preserve the shared-row structure is the obvious alternative and was not tested.

A second flaw in my own sweep

dep_c is a noise-mixing coefficient, not ξ. At dep_c=0.20 the per-cell ξ is only 0.021. So the sweep never entered the decision-relevant regime around ξ≈0.45, and the "90% power at ξ≈0.094" figure comes from the single top level. The shape of the comparison is sound; the resolution is coarse.

Status

QUANTIFIED, not FIXED. The substantive finding — that permutation addresses the max and not the threshold — is established and is a real refinement of the research's c.3 recommendation. But the size validity is not cleanly established, and the permutation scheme carries an unresolved design question. Adopting this as a gate needs both settled first, plus pre-registration of α and B.

Nothing was changed. No dial, band or threshold touched.

What a corrected run needs

Permute cells jointly (shuffle the master series once, re-cut the windows) so the reference preserves the cross-cell dependence, and compare size against the independent-permutation scheme.
Parameterise the sweep by measured per-cell ξ, not by the mixing coefficient, so the decision-relevant band around 0.45 is actually covered.
More size reps — 120 gives se 2.0%, which cannot distinguish a 2σ excursion from a real defect.
n=500 per cell here, reduced from 1,000 after a 10-minute timeout. Per-cell ξ noise scales as n^−½, so the null floor (0.065) is wider than the cascade's real n=3,990 would give.

Evidence: harness/permutation_reference_for_max_evidence.json. Reproduce: VIRTUAL_ENV="" uv run --python 3.13 --no-project --with numpy --with pandas python3 harness/permutation_reference_for_max.py (~7 min)