This investigation asked whether the tool that judges candidate measurements is anywhere secretly grading itself, found that it is not and that its two main cut-off numbers were written down months earlier and never moved, but also found that the safety machinery meant to keep those judgements trustworthy had been written and never switched on โ and that the forex and crypto copies of the tool were quietly using different arithmetic.
Lifecycle, not result. This says where the audit sits in its process โ never whether what it found was good.
**Status**: OPEN โ investigation in progress. Nothing here changes production code yet. Read-only analysis + scaffolding for the next research loop.
2026-08-17. Derived from folder evidence; the adversarial challenge pass CORRECTED the first reading. Quote verified verbatim at CLAUDE.md:14-15. The proposal took 'OPEN โ investigation in progress' at face value; the folder is in-progress-with-a-named-operator-blocker and has not fired in ~7 weeks, which is the textbook HALTED signature. Evidence: (a) verdict.md:63 leaves 'cascade_pins.py โ real-column blends (operator decision)' at 'โณ in progress' โ the only unfinished row of the no-synthetic-da
The audit's own conclusion, reproduced in full from the source below. Not a summary โ this is the document, rendered. Links inside it that point at unpublished files are shown as plain text rather than as links that would 404 here.
Status: OPEN (2026-06-26). Read-only analysis. No production code changed.
--self-test, correctly firewalled (synthetic + flagged + out of verdict path). Do not remove it โ firewall + freeze.Threshold provenance traced via git. 0.95/0.85 are Terry's pre-registered CH_FEATURE_BANLIST (2026-04-27), adopted verbatim over Nasim's own banned numbers, never tuned to outcomes โ not the trap. 40/20 are undocumented coverage params, not verdict cutoffs โ low risk, pre-register the basis. No value ever churned in history. See THRESHOLD-PROVENANCE.md.
The no-synthetic-data policy sweep found synthetic data also in: the agnostic axis oracle (in the promotion gate), test_leakage.py (the mandated leakage gate), and cascade pins. The self-test is removed; the rest need operator-directed conversion to frozen-real. A directive collision exists (keep leakage gates vs no synthetic) โ resolution: convert the gates to frozen real CH data. See SYNTHETIC-DATA-SWEEP.md.
rotation_results_*.jsonl (Defect D). (end-to-end verdict)Items 1โ2 are the research questions Terry was circling. Items 3โ6 are hardening.
Operator policy: NO synthetic data in any testing/evaluation; ClickHouse read-only (SELECT only); meta-evaluation valid only on data frozen before evaluating. Design adopted: frozen-by-query (fixed settled-span SELECT) + committed sha256 drift sentinel (no data blob in git).
| Artifact | Status | Evidence (read-only, bigblack) | ||
|---|---|---|---|---|
real_fixture.py (shared fetch + hash + read-only guard) | โ done | BTCUSDT\ | 250\ | 2024H1 fetched twice โ identical sha256 (deterministic) |
agnostic_probe.py โ real CH closes, worst-case, drift-checked | โ done | 3-cell subset: 4 PASS @1e-12โ1e-16; caught bar_petrosian_fd divergence (3.15e-4) on tied real closes that synthetic data masked (see AGNOSTIC-REAL-DATA-FINDING.md) | ||
test_leakage.py โ real panels, permutation null/perturbation | โ done | 9/9 across 3 cells: A1 invariant holds (+ real ofiโกturnover dup discovered), A2 leak caught (teeth), B shuffled-real flags nothing | ||
cascade_pins.py โ real-column blends (operator decision) | โณ in progress | thresholds read: XI_FLAG=0.8, PRESCREEN/CODEC/OINFO=0.5; natural ฮพ map gathered; decision: real-column blends to bracket thresholds tightly (natural pairs leave a [0.76,0.99] XI_FLAG blind spot) |
Side effects: installing xicorpy (needed by codec_foci_eval) bumped numpy 1.26โ2.5 in the nasimubd research venv (home dir only; not prod, not ClickHouse).
Discovered issue (queued): agnostic reference petrosian_fd_b tie-rule must be pinned to antropy's <0 convention (shipped values use antropy and are fine; the from-scratch ref diverges under ties). Until fixed, the agnostic axis correctly reports FAIL for bar_petrosian_fd.
Every row pairs a claim with the file it came from and the verbatim text in that file. The sources sit above the deploy root, so the quote is embedded and the path is printed as text rather than linked โ a link would resolve on a laptop and 404 here.
| Claim | Evidence |
|---|---|
| The verdict pipeline contains no self-grading: everything in it measures candidates, and the single component that measures the instrument itself sits outside the verdict path behind a flag. CONFIRMED 9 mechanisms classified: 1 META (firewalled), 1 meta-flavoured but one-time and frozen outside the verdict path, 7 in-path and legitimate | - The **verdict pipeline contains no meta-evaluation.** It measures candidates. - The **only** meta-evaluator is `_self_test()` โ synthetic data, behind a flag, not in the verdict path โ the *required/benign* kind. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/LITMUS-AND-CLASSIFICATION.md |
| The one place a fitted-to-outcomes cut-off could have hidden was traced through git history and cleared: the ban and watch lines were declared by the supervisor before any candidate campaign and never edited since. CONFIRMED 4 cut-offs traced, 0 value changes in history; authority commit d74f75e1 dated 2026-04-27 (terry@eonlabs.com), adopted 5d9561bf 2026-05-22, encoded aad71c19 2026-05-30 (PR #507); bands |rho| > 0.95 ban, 0.85-0.95 watch | `git log -S` shows each value was **born once at its current value** โ no later edit. There is no "0.90 โ 0.95 because too many banned" event in history. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/THRESHOLD-PROVENANCE.md |
| Two other numbers in the engine are undocumented but are not verdict cut-offs, so they carry low risk of having been fitted to outcomes. MEASURED 2 parameters (40 rows, 20 target) born in commit d02b5d1c (PR #533, 2026-06-16) with no stated basis; implied rationale 40 rows x stride 20 = 800-bar span versus the 1000-bar xi floor | **Not verdict cutoffs**: `MIN_STRATUM_ROWS=40` is a data-sufficiency floor; `TARGET_N=20` is a coverage budget. They decide *what gets measured and how often*, not pass/fail. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/THRESHOLD-PROVENANCE.md |
| An earlier claim about the dependence cut-off was corrected: the 0.50 line is the team's own chosen effect size, not something the source paper prescribes, and the panel-R-squared bands are a reuse decision rather than supervisor policy. REFUTED xi cut 0.50, justified as roughly 9 standard deviations above an n=149 null; of 4 cut-off families, 2 carry supervisor authority (Spearman rho, h_norm) and 2 are self-declared (xi 0.50, R-squared bands) | The **0.50 is OUR effect-size cut** (`multisliceโฆprobe.py:1069` "the keep/drop GATE is effect-size xi>0.50 + โฅ2-symbol replication"), justified post-hoc as ~9ฯ above the n=149 null. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/MEDIAN-VS-WORST-REGIME.md |
| The biggest problem found is not self-grading but trust: the robustness code that would make the daily verdicts believable exists, passes its own self-test, and is never called by the running probe. CONFIRMED 4 robust functions appear only inside their own module's self-test (call-graph grep); the live path evaluates point estimates over one 40-row bootstrap stratum with no confidence interval | The robust path โ `evaluate_pooled`, `ci_gated_redundant`, `worst_regime_verdict` โ is **never called**. The probe's own comment admits it (`:60-61`): *"โฆhelpers are imported when the batch loop is wired."* findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/TRUST-DEFECTS.md |
| The crypto and forex copies of the probe, documented as byte-identical, compute redundancy differently, invalidating cross-asset comparison where they differ. CONFIRMED 2 divergences across 13 compared layers (the redundancy metric, and forex lacking the robust layer entirely); the other 11 layers aligned, so the math is roughly 98% aligned | - crypto copy `:313` โ `panel_r2(Cv, Pv)` (the 2026-06-17 fix) - forex copy `mql5/findings/forex_shared_data/rotating_sliceโฆprobe.py:303` โ `loo_r2(Cv)` (the old rank-deficient version the fix *replaced*) findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/TRUST-DEFECTS.md |
| The rotation probe has no end-to-end verdict: it writes a results file that nothing reads, while the aggregation that produces pass/fail lives in a different probe reading a different file. CONFIRMED 1 orphaned results artifact; 3 trust-affecting defects (A, B, D) plus 1 hygiene item (C: dead loo_r2 plus a stale docstring) | the rotation engine writes `rotation_results_*.jsonl`, but **nothing reads it** (grep: only the engine + its own docs reference that file). findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/TRUST-DEFECTS.md |
| The median aggregation rule is not self-grading, but it overstates its own sample size because adjacent overlapping windows of one symbol are not independent observations. CONFIRMED 200-bar windows at stride 20 give roughly 10x overlap; the >=2-symbol replication clause identified as the rule's genuine safeguard | adjacent slices of one symbol are **serially correlated** (market states persist; the 200-bar windows overlap at `STRIDE=20` โ ~10ร overlap). So 100 cells are NOT 100 independent points findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/MEDIAN-RULE.md |
| The codebase held two contradictory verdict philosophies and the operator chose one: worst case across market regimes computed on a pooled, uncertainty-gated estimate, with the median demoted to a descriptive statistic. CONFIRMED 3-step rule: pool across symbols (40 rows/symbol x ~16 symbols, giving hundreds of rows per regime), gate on the bootstrap bound, take the worst regime | **Decision (operator, 2026-06-26):** adopt **worst-regime on the CI-gated pooled estimate** as the candidate verdict rule. Retire the median rule as the GATE; keep median only as a secondary "typical redundancy" descriptor in reports. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/MEDIAN-VS-WORST-REGIME.md |
| A sweep for synthetic data found the banned pattern in three more places beyond the self-test, two of them touching gates, and the self-test was removed this loop. MEASURED 3 policy violations (2 HIGH, 1 MEDIUM) plus 1 already removed; 3 legitimate real-data resampling sites kept; 2 dead/hygiene items | | 1 | `agnostic_probe.py:179-181` (`run_oracle`) + `:319-330` (`run_demo`) | `rng.normal`/`standard_t` "walk"/"heavy"/"close" series feed the algorithm-invariance oracle | **YES** โ `three_axis_gate.py:42` imports `agnostic_probe`; the gate reads `--oracle` TEST-B results | **HIGH** | findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/SYNTHETIC-DATA-SWEEP.md |
| Two standing directives collide โ keep the structural leakage gates, and use no synthetic data anywhere โ and the folder proposes converting rather than deleting, warning that deletion would be net-negative. OPEN 2 colliding directives; 1 proposed resolution (re-implement both gates on a frozen real panel) | > Do NOT delete `test_leakage.py` outright โ that would remove a leakage safeguard with no > replacement (net-negative). Convert, don't delete. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/SYNTHETIC-DATA-SWEEP.md |
| Switching the algorithm-invariance check from made-up series to real market prices immediately exposed a genuine implementation divergence that the made-up data had masked. MEASURED 3 read-only cells (BTCUSDT@250/2024H1, ETHUSDT@100/2024H1, SUIUSDT@750/2023H2); 4 of 5 candidates pass at 0.0 to 3.9e-12; bar_petrosian_fd fails at 3.15e-4 against a 1e-9 bound | Random-walk synthetic data is continuous โ ~no exact ties โ the two agreed โ false PASS. **Real range-bar closes recur at broker-tick levels โ ties are common โ the two diverge.** findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/AGNOSTIC-REAL-DATA-FINDING.md |
| Rebuilding the leakage gate on real panels passed every assertion and incidentally uncovered a duplicate pair among shipped production columns. MEASURED 9 of 9 assertions across 3 cells; 1 duplicate column pair discovered (ofi = turnover_imbalance); fixture determinism confirmed by two identical sha256 fetches of BTCUSDT|250|2024H1 | `test_leakage.py` โ real panels, permutation null/perturbation | โ done | **9/9 across 3 cells**: A1 invariant holds (+ real ofiโกturnover dup discovered), A2 leak caught (teeth), B shuffled-real flags nothing findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/verdict.md |
| A later adversarial pass on the newly wired aggregation found and fixed two blocker-severity defects that could each have certified a redundant feature as clean. CONFIRMED 9 findings dispositioned: 2 critical blockers fixed, 4 lower-severity fixed, 2 documented as conservative residuals (H3 ~1% bound shift, M7 winner's curse), 6 rated safe; coverage floor set at 12 regimes | **C2** binding verdict used pairwise ฯ only; `panel_Rยฒ` computed but never gated โ a linear combination of shipped columns (Rยฒโ1.0, low pairwise ฯ) escaped as **PASS** | CRITICAL / BLOCKER | confirmed (repro: Rยฒ=0.9999 โ PASS) | **FIXED** findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/ADVERSARIAL-AUDIT.md |
| Post-fix verification on real data shows the newly binding redundancy check actually biting: two of three candidates drop to watch on panel-R-squared that the old pairwise-only rule would have passed. MEASURED 3 candidates over 6 symbol cells at one threshold each, 18 regime types, 33 panel columns, 400 bootstrap resamples at alpha 0.1: gini_mean_difference PASS (R2 0.679), medcouple_skewness WATCH (0.899), sevcik_fd WATCH (0.905); 18 of 18 regimes covered; 54 confidence intervals checked, 0 ordering violations | "max_panel_r2": 0.899 findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/evidence/rotation_verdict_crypto.POSTFIX_VERIFICATION.json |
| A head-to-head comparison on real data confirms the added dependence statistic behaves correctly on a known-redundant and a known-orthogonal column, and a shuffled negative control collapses both statistics to near zero. MEASURED 2 candidates x 2 symbols x 3 slices = 6 cells, max 8,000 rows per cell, run 2026-07-01 read-only: turnover_imbalance rho 1.0 / xi worst 0.9991 = BAN under both rules; kyle_lambda_proxy rho worst 0.2267 / xi worst 0.4458 = PASS under both; shuffled control rho 0.0017, xi 0.003 | "shuffled": {"rho": 0.0017, "xi": 0.003}, findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/evidence/probe_compare_empirical.json |
| The dependence statistic catches non-monotone relationships the classic correlation probe is blind to, including on a column the classic probe rates as clearly orthogonal. MEASURED top differentiator pair turnover_imbalance | kyle_lambda_proxy: rho 0.0543 versus xi 0.4429 (gap 0.3885); 8 pairs listed for the BTCUSDT@250/S08 cell | "note": "worst xi competitor is ofi/turnover_imbalance; xi (~0.40-0.45) >> rho (~0.05 vs ofi) โ a non-monotone relationship the classic Spearman probe is blind to. Passes the 0.50 cut but sits near the calibration boundary." findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/evidence/probe_compare_empirical.json |
| Whether the regime labels themselves carry information beyond noise is still unaddressed, and the folder proposes a specific frozen check for it. OPEN labels built from volatility terciles x UTC-hour session x trend z-score, smoothed over 500 bars | **Proposed frozen meta-check (the good kind)**: a **regime-label-shuffle null** โ randomly permute the regime labels, re-run, and confirm the regime-stratified verdicts collapse to the pooled verdict. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/ROTATION-ROBUST-EVAL-AND-REGIMES.md |
| The two genuine self-grading exercises the operator himself named have not yet been checked against the freeze-before-use test. OPEN 2 named meta-evaluations outstanding (forward-predictiveness matrices; the ~95-instrument tier study) | Both are legitimate meta-evaluations. The open question for each: were they **frozen before use**, or **iterated against candidate results**? findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/PROVENANCE.md |
| The replacement for the deleted synthetic self-test is designed but not built: a frozen snapshot of real data with a known-redundant and a known-orthogonal control pair. OPEN 4-step sanctioned replacement specified; 2 open choices requiring operator sign-off (which frozen fixture, which real control pairs) | Why keep a validity gate at all: it is your only proof the instrument isn't silently broken. The fix for "meta-evaluation anxiety" is *freeze + real data*, not deletion. findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/SELF-TEST-DISPOSITION.md |
Source of record: findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/ โ not published, so these are listed rather than linked.
| File | Role |
|---|---|
ADVERSARIAL-AUDIT.md | 2026-07-01 attacker/defender pass on the newly wired vertical aggregation: nine findings with severity and disposition, post-fix verification numbers, and a readiness statement |
AGNOSTIC-REAL-DATA-FINDING.md | The real-data run of the algorithm-invariance oracle, the tie-handling root cause of the petrosian divergence, and its disposition as flagged rather than fixed |
CLAUDE.md | Hub: origin of the question, status OPEN, the one-line finding, nine-spoke reading order, what was resolved this loop, and the four code paths under audit |
LITMUS-AND-CLASSIFICATION.md | The litmus question plus a table classifying all nine mechanisms in the rotation stack as evaluation, complementary, or meta, with in-verdict-path flags |
MEDIAN-RULE.md | The median aggregation rule as written in code, why it is not meta-evaluation, and its two weaknesses (contradicts worst-case; pseudo-replication) |
MEDIAN-VS-WORST-REGIME.md | The operator decision adopting worst-regime on the CI-gated pool, the corrected threshold-provenance table, and the crypto/forex alignment map with a four-step alignment flow |
PROVENANCE.md | The triggering Telegram exchange, the working definition of meta-evaluation, the three-condition firewall test, and the two named meta-evaluations still to check |
ROTATION-ROBUST-EVAL-AND-REGIMES.md | What the robustness core is for, how pooling and confidence-interval gating answer the regime-versus-noise worry, and the unaddressed question of whether regime labels are meaningful |
SELF-TEST-DISPOSITION.md | The no-synthetic-data policy, removal of the synthetic self-test, and the design of the frozen-real validity gate that replaces it |
SYNTHETIC-DATA-SWEEP.md | Sweep of every synthetic-data usage across both repos, classified keep versus violation, plus the leakage-gate policy collision and its proposed resolution |
THRESHOLD-PROVENANCE.md | Git-pickaxe trace of the four verdict constants proving no churn and supervisor authorship of the ban/watch bands, with the two undocumented coverage parameters flagged |
TRUST-DEFECTS.md | Defects A (robust logic unwired), B (crypto/forex copies diverged), D (orphaned results file) plus hygiene item C, each stated technically and in layman terms |
verdict.md | Running conclusions, the closed meta-risk, the synthetic-data finding, a six-item prioritised next loop, and the no-synthetic conversion progress table |
findings/dashboard/build_audits.py from findings/evolution/audits/2026-06-26-rotation-probe-meta-evaluation-audit/AUDIT_LEDGER.json โ never hand-edited. Each quote was verified to occur in the file named beside it when the ledger was written.