A survey of about 95 published ways to measure whether one market feature is really telling you the same thing as another, each checked for whether usable open-source code exists โ followed by test runs on real Bitcoin data showing that a cheap rank-based method detects genuine curved relationships in 118 of 171 feature pairs that the project's current correlation check calls independent.
Lifecycle, not result. This says where the audit sits in its process โ never whether what it found was good.
**Status:** **Phase 1 (catalog) COMPLETE.** ~95 candidate measurement instruments documented across 3 tiers ร 8 categories with FOSS status. Phase 2 (empirical spike on real data) is OUT OF SCOPE for this folder.
2026-08-17. Derived from folder evidence; the adversarial challenge pass CORRECTED the first reading. Quote verified verbatim at verdict.md:4, but the proposal rests DONE on a verdict file that the folder itself has outgrown. That verdict (dated 2026-05-16) asserts 'Phase 2 (empirical spike on real data) is OUT OF SCOPE for this folder' and lists 'candidates/ โ per-metric deep dives (empty initially)'. Both are false on disk: candidates/ contains six Phase-2 spike subfolders (chatterjee-xi, chatte
The audit's own conclusion, reproduced in full from the source below. Not a summary โ this is the document, rendered. Links inside it that point at unpublished files are shown as plain text rather than as links that would 404 here.
Date: 2026-05-16 Status: Phase 1 (catalog) COMPLETE. ~95 candidate measurement instruments documented across 3 tiers ร 8 categories with FOSS status. Phase 2 (empirical spike on real data) is OUT OF SCOPE for this folder.
This audit closes the following deliverable:
> A defensible baseline of state-of-the-art instruments to measure feature orthogonality in financial time series forecasting, distinct from what's already in production, with FOSS implementations identified per candidate.
Reproducible from this folder:
RESEARCH-DIGEST.md โ canonical ~95-metric catalog (3 tiers ร 8 categories)METRIC-EVALUATION-PROTOCOL.md โ three-tier framing methodologyBASELINE-RULED-OUT.md โ 5 production matrices + 5 gaps + 4 duplicates markedVERIFICATION-VERDICT.md โ per-metric FOSS + license + complexity + vaporware warningscategories/ โ 8 per-category breakdowns (A1-A4, B1-B2, C1-C2)implementations.jsonl, papers.jsonl, search_log.jsonl โ structured corpus (litscan format)| Dimension | Status |
|---|---|
| Coverage of pairwise dependence (Tier A1) | ~25 metrics catalogued; top picks Chatterjee ฮพโ + dCor + Copula Entropy |
| Coverage of conditional dependence (Tier A2) | ~10 metrics catalogued; top pick CODEC/FOCI |
| Coverage of higher-order / synergy (Tier A3) | ~8 metrics catalogued; top pick O-information |
| Coverage of matrix decomposition (Tier A4) | ~12 metrics catalogued; top pick Marchenko-Pastur + RIE |
| Coverage of causal direction (Tier B1) | ~15 metrics catalogued; top pick Reduced TE (Kirkley 2025) |
| Coverage of regime stability (Tier B2) | ~7 metrics catalogued; top pick Sliced Wasserstein dependency |
| Coverage of FDR-controlled selection (Tier C1) | ~12 metrics catalogued; top picks DeepLINK-T + e-BH |
| Coverage of finance-specific frameworks (Tier C2) | ~6 frameworks catalogued; top pick Lรณpez de Prado cMDA + HRP |
| FOSS verification per metric | Done โ library + license + URL + maintained status |
| Distinctness from production baseline | 4 duplicates explicitly marked in BASELINE-RULED-OUT.md |
| Distinctness from sibling 64-candidate audit | Confirmed โ different object (instruments vs features); no overlap |
These strengthen the catalog or move toward Phase 2 but are explicitly NOT closure-blocking for the research-catalog deliverable.
| Item | Why deferred | Where it lives |
|---|---|---|
| Phase 2 โ empirical spike of top picks on real bigblack data | Catalog is research input, not finding. Spike requires substrate decision (which threshold/symbol), test harness, hold-out validation | Separate audit folder when Phase 2 begins |
Per-metric deep-dive files in candidates/ | Only filled when a metric advances to Phase 2 | candidates/_template.md ready for use |
| Closing the 14+ missing categories (TDA, signatures, self-supervised, graph-based, manifold learning, tensor decomposition, Bayesian comparison, quantum, etc.) | One more brute-force round (~30 min) would close coverage gaps the user identified | Future expansion round |
| Empirical FOSS validation (install + run on toy data) | Beyond catalog scope; required before spike commitment | Phase 2 task |
| License-conflict analysis for downstream integration | Tier-by-tier; requires knowing the integration target (which sub-system) | Per-spike basis in Phase 2 |
Dashboard surfacing at findings/dashboard/orthogonality/measurement-instruments/ | Stage-2 HTML rendering; content needs to stabilize first | Future, after catalog content settles |
Append entry to findings/evolution/audits/CLAUDE.md index | This file already lists the prior peer audit; needs new entry for this audit | Done in same commit as catalog creation |
Append entry to evolution.jsonl (if/when this becomes a finding) | Catalog is INPUT, not a finding. An evolution.jsonl entry would record outcomes from spiking (e.g., "dCor flagged 3 pairs Pearson missed; held on hold-out") | Phase 2 deliverable |
implementations.jsonl but NOT used as a gating filter. The catalog is empirical-evidence-first; adoption decisions are deferred to Phase 2.METRIC-EVALUATION-PROTOCOL.md and categories/B1-causal-direction.md.findings/evolution/audits/2026-04-29-evolutionary-gp-multi-asset/RESEARCH_overnight_synthesis_round2.md:39 (sister repo) and sat unactioned for 15 days. The catalog explicitly avoids producing review-blocking artifacts; everything is markdown + JSONL with no senior-review gates.Operator decides. Three plausible paths:
The catalog is at a defensible Phase 1 stopping point. No path is forced.
CLAUDE.md โ audit hubRESEARCH-DIGEST.md โ canonical catalogMETRIC-EVALUATION-PROTOCOL.md โ three-tier framing methodologyBASELINE-RULED-OUT.md โ production baseline + duplicatesVERIFICATION-VERDICT.md โ FOSS verificationcategories/ โ per-category breakdownscandidates/ โ per-metric deep dives (empty initially)../2026-05-08-orthogonal-features-btcusd/Every row pairs a claim with the file it came from and the verbatim text in that file. The sources sit above the deploy root, so the quote is embedded and the path is printed as text rather than linked โ a link would resolve on a laptop and 404 here.
| Claim | Evidence |
|---|---|
| The catalogue holds roughly 95 measurement methods sorted into three tiers and eight categories, with about half having maintained packaged code. MEASURED ~95 total: Tier A ~55, Tier B ~22, Tier C ~18; ~50 fully FOSS, ~25 code-released-not-packaged, ~20 with no code found | | Fully FOSS (PyPI/CRAN, maintained) | ~50 (permissive MIT/BSD/Apache ~30; GPL ~20) | findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/RESEARCH-DIGEST.md |
| Almost none of the catalogue overlaps with what the project already runs โ only four entries are duplicates of the deployed baseline. CONFIRMED 4 duplicates of ~95; 5 production matrices plus 2 adjacent probes documented as the baseline | **Total duplicates: 4 of ~95 metrics.** The remaining ~91 entries are distinct from the production baseline. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/BASELINE-RULED-OUT.md |
| The deployed baseline is blind to five specific kinds of duplication, which is the stated reason the catalogue exists. CONFIRMED 5 named gaps: non-linear redundancy, conditional redundancy, causal direction, synergy, regime stability | Baseline measures: linear/monotone bivariate redundancy (Pearson, Spearman) + univariate spread (h_norm) + featureโtarget MI (Tier-2). It does NOT detect: findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/BASELINE-RULED-OUT.md |
| Anything reaching production from this catalogue is explicitly gated on a human review that has not happened. ASSERTED 4-stage protocol; stage 4 (Deployment) blocked | **Status:** Provisional. Part I and Part II are operationally grounded. Deployment stage is **๐ BLOCKED awaiting Terry's review** โ do not ship a candidate to production until that review lands. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/METRIC-EVALUATION-PROTOCOL.md |
| A first cheap test of the rank-based method reproduced every known-duplicate pair and flagged three pairs the current correlation check considers independent. MEASURED 8 pairs ร 2 directions in 0.03 s, N=1,000 per pair; ฮพ 0.997 on the perfect duplicate; ฮพ 0.544/0.536, 0.716/0.746 and 0.613/0.577 on the three pairs Spearman calls near-zero | **PROCEED TO TIER 1.** Plus a **significant cross-candidate corroboration** of the HSICAgg pair-8 finding. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/chatterjee-xi/gate-0.5-verdict.md |
| A kernel-based method run on the same eight pairs agreed, but its result was capped by how many permutations were run. MEASURED 21.2 s wall clock, 4 bandwidths ร 200 permutations ร 8 pairs; 7 of 8 pairs at the p=0.0050 floor | **Resolution floor:** at 200 permutations, the smallest p-value possible is 1/(200+1) โ 0.0050. Multiple pairs hitting exactly this value means HSICAgg detects dependence at or below the resolution limit (would benefit from more permutations to differentiate strengths). findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/aggregated-hsic/gate-0.5-verdict.md |
| A surrogate-data calibration then showed the surprising finding was real signal rather than an artefact of the data's own memory โ and that one of the two methods over-flags on financial data. CONFIRMED Chatterjee observed 0.613 vs null max 0.052 (11.8ร), empirical p=0.0000 over 500 surrogates; HSICAgg empirical Type-I 10โ13% against a nominal 5% (2โ2.7ร inflation); 3.3 min total | **The pair-8 finding (`kyle_lambda_proxy โ buy_volume` non-linear dependence) is REAL, not autocorrelation artifact.** This is decisively established by Chatterjee ฮพโ and corroborated (borderline-positive) by HSICAgg. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/iaaft-calibration/gate-3-verdict.md |
| Widening the test to every pair in the panel showed the finding is the norm, not an exception: two thirds of pairs look independent to the current check yet carry detectable curved dependence. MEASURED 171 pairs from 19 columns, N=1,000, 69 s; 171/171 significant at q=0.10; 118/171 (69%) Spearman-independent yet flagged; only 11/171 (6.4%) flagged by the production |ฯ| โฅ 0.95 rule | **STRONG SIGNAL โ the 8-pair finding is the rule, not the exception.** Across all 171 microstructure feature pairs in our broadening surface, **100% are Chatterjee BH-FDR-significant at q=0.10**, and **69% (118 of 171) are simultaneously Spearman-independent (|ฯ| < 0.30) yet Chatterjee BH-FDR-significant**. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/chatterjee-broadening/verdict.md |
| The audit refuses to read that result as an instruction to delete features, and names the alternative explanation. CONFIRMED 19 of ~53 production features testable at the 100 dbps threshold; 36 were previously classed insufficient-sparse | โ ๏ธ **"100% of pairs are significant" does NOT mean "100% of pairs should be dropped from a feature set."** Significance โ utility. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/chatterjee-broadening/verdict.md |
| The finding held on two other coins, but the effect shrinks sharply as the coin gets smaller. CONFIRMED 3 symbols ร 5,000 bars, 17 s; 6/6 pair-symbol combinations validated, observed/null-max ratios 4.3โ11ร; ฮพ falls 0.652 (BTC) โ 0.164 (ETH) โ 0.079 (ADA) on the headline pair | **REPLICATED with diminishing magnitude on smaller-cap symbols.** The kyle_lambda_proxy non-linear dependence finding is **IAAFT-validated REAL on all three symbols (BTC, ETH, ADA)**, but the **magnitude** of the dependence varies substantially: it is strongest on BTC and weakens monotonically through ETH to ADA. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/multi-symbol-robustness/verdict.md |
| Re-running everything on a different machine against a fresh data pull reproduced six of seven checkpoints; the seventh was cut off by a compute budget, not by a changed result. CONFIRMED 6/7 checkpoints matched; broadening 70.2% vs 69.0% Pearson-blind; headline ฮพ 0.647 vs 0.613; Type-I 10.0% identical on both machines | **Match rate: 6 of 7 (85.7%) exact or qualitative match. 1 of 7 incomplete due to compute budget.** findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/bigblack-replication/verdict.md |
| The later formal selection gate for the conditional-dependence cascade came back only partly successful: the method reproduces existing labels but was never shown to add anything the current baseline does not already give. MEASURED 325 cells; mutation-oracle kill rate 0.9606 over 127 mutants with 0 boundary survivors; mean walk-forward Jaccard 0.8811 (min over epochs 0.8523); 2 of 3 cascade stages have no faithful analytic null; 7 headline pairs, ฮพ 0.97โ0.999 or 0.02โ0.17 with nothing in between | > **Honest ceiling.** Selection-partial โ ฮพ REPRODUCES the Pearson-derived duplicate/orthogonal labels with stable, leakage-free walk-forward Jaccard (S1/S2-ฮพ/S3/S5-ฮพ machinery PASS). NO incremental value over Pearson/Spearman is demonstrated (S4 deferred โ circular labels) findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/selection_verdict.md |
| A successor pre-registration was frozen because the earlier numbers had been produced by moving thresholds after seeing the data โ so those gates must be re-measured before they count. OPEN Frozen grid: 6 symbols ร 3 thresholds ร 7 regime slices; caps 10 cores / 5 GB / 5400 s; pass rules fixed (Type-I โค 0.06, Jaccard โฅ 0.20 in every regime) | **Status:** FROZEN 2026-06-07, before any re-measure under this spec. Scope = **promotion verdict for GCM (A2) + O-information (A3) on top of the production orthogonality baseline.** findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/gcm-oinfo-selection-s0.md |
| The candidates folder's own index still describes itself as empty even though five spike folders with real results sit inside it. REFUTED 5 populated spike subfolders present (aggregated-hsic, chatterjee-xi, iaaft-calibration, chatterjee-broadening, multi-symbol-robustness) plus a bigblack replication | Currently **empty** (apart from the template) because Phase 1 (research catalogue) does not produce per-metric spike outputs โ that's deferred to Phase 2 with substrate decisions and empirical validation. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/CLAUDE.md |
| The catalogue names the whole research areas it did not search, rather than presenting itself as complete. CONFIRMED 10 named missing categories including topological data analysis, path signatures, self-supervised decorrelation, tensor decomposition | A future round of brute-force agents could close these gaps. The current catalog reflects the buckets we asked for; it is not an exhaustive taxonomy of feature-dependence measurement. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/RESEARCH-DIGEST.md |
| The audit deliberately avoids producing anything that needs sign-off, citing a prior adopt-verdict that sat unactioned. CONFIRMED 15 days unactioned | 7. **A written "Adopt" verdict is not adoption.** Group Knockoffs has an explicit verbatim "Adopt Phase 1" in `findings/evolution/audits/2026-04-29-evolutionary-gp-multi-asset/RESEARCH_overnight_synthesis_round2.md:39` (sister repo) and sat unactioned for 15 days. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/verdict.md |
| The audit corrects its own earlier framing: the project's three-axis feature rubric was wrongly stretched to cover measurement tools. CONFIRMED | 4. **The 3-axis rubric (Orthogonal ร Parameterless ร Agnostic) is for features, not matrices.** AI extension of the rubric to measurement instruments was unsanctioned. Catalog uses the three-tier framing instead โ explicitly not the 3-axis rubric. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/verdict.md |
| The data layer this audit's spikes originally used has been moved out of the folder and the old path is a redirect stub. CONFIRMED Superseded 2026-05-23 in PR #501; substrate grew from 1 symbol ร 5,000 rows to ~552 parquet slices | **Status:** This folder is **superseded** by the top-level shared data layer at [`/findings/evolution/shared_data/`](../../../../shared_data/). Do not add new files here. findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/candidates/shared_data/CLAUDE.md |
Source of record: findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/ โ not published, so these are listed rather than linked.
| File | Role |
|---|---|
BASELINE-RULED-OUT.md | What is already deployed: 5 production matrices, the 5 gaps they miss, the operational constraints, and the 4 catalogue duplicates. |
CLAUDE.md | Audit hub โ scope, four-agent gathering method, tier framing, file index, recommended reading order. |
METRIC-EVALUATION-PROTOCOL.md | Two-part protocol: tier-assignment test for organising the catalogue, and the 4-stage validation ladder whose final stage is blocked. |
OPERATOR-GUIDE.md | Plain-English companion to the protocol, with an FAQ on the headline candidate metrics. |
RESEARCH-DIGEST.md | Canonical catalogue: aggregate counts, one top pick per gap, tier ร category breakdown, vaporware warnings, coverage gaps. |
SELECTION-GATE-DESIGN-protocol-lens.md | Design of the S1โS5 selection gate ladder, treating the cascade rather than a single metric as the unit under test. |
VERIFICATION-VERDICT.md | Per-metric library, licence, URL and maintenance status, with aggregate FOSS and licence distribution. |
candidates/CLAUDE.md | Per-metric folder hub; still describes the folder as empty although five spike folders exist. |
candidates/_template.md | Template for a per-metric deep dive. |
candidates/aggregated-hsic/gate-0.5-verdict.md | Result of that spike: all known-redundant pairs flagged, one disagreement with the baseline, permutation resolution floor noted. |
candidates/aggregated-hsic/pre-spec.md | Locked hypothesis, substrate and drop rules for the kernel-based method's first spike. |
candidates/bigblack-replication/diff_report.md | Numeric diff between the remote-host and laptop runs, spike by spike. |
candidates/bigblack-replication/pre-spec.md | Locked plan for re-running all five spikes on the remote host against a fresh data pull. |
candidates/bigblack-replication/verdict.md | Cross-platform replication scoreboard and the operational lessons, including a lost-results checkpointing defect. |
candidates/chatterjee-broadening/pre-spec.md | Locked plan and decision rule for extending the test to every pair in the panel. |
candidates/chatterjee-broadening/verdict.md | Panel-wide result with two competing readings of the 100%-significant outcome, left for the operator to choose. |
candidates/chatterjee-xi/gate-0.5-verdict.md | Result of that spike: agreement on known duplicates, independent corroboration of the surprising pair, ~700ร cheaper. |
candidates/chatterjee-xi/pre-spec.md | Locked hypothesis and identical pair list for the rank-based method, for an apples-to-apples comparison. |
candidates/codec-foci-selection-s0.md | Frozen S0 pre-registration for the CODEC/FOCI cascade, including the hold-out Type-I pair set. |
candidates/codec-foci.md | Per-metric deep dive on the CODEC/FOCI cascade with its frozen exploration pre-spec and the three-gap rationale. |
candidates/gcm-oinfo-selection-s0.md | Frozen S0 pre-registration for the successor GCM + O-information run; supersedes the CODEC config and requires a re-measure. |
candidates/iaaft-calibration/gate-3-verdict.md | Calibration result: the finding is real, and the kernel method's false-positive rate is inflated 2โ3ร on financial data. |
candidates/iaaft-calibration/pre-spec.md | Locked plan for the surrogate-data calibration that separates real signal from autocorrelation artefact. |
candidates/multi-symbol-robustness/pre-spec.md | Locked plan and replication rule for testing the finding on two additional coins. |
candidates/multi-symbol-robustness/verdict.md | Replication result with the liquidity-gated diminishing-magnitude pattern. |
candidates/selection_verdict.md | Selection-gate verdict for the three-sieve cascade: machinery passes, incremental value not demonstrated, deployment blocked. |
candidates/shared_data/CLAUDE.md | Redirect stub โ the audit-internal data layer was promoted to a top-level shared layer in PR #501. |
categories/A1-pairwise-dependence.md | Category A1 โ pairwise dependence measures including mutual-information estimators; the largest category. |
categories/A2-conditional-dependence.md | Category A2 โ conditional dependence measures; top pick CODEC/FOCI. |
categories/A3-higher-order-synergy.md | Category A3 โ higher-order and synergy measures; top pick O-information. |
categories/A4-matrix-decomposition.md | Category A4 โ matrix, spectral and random-matrix-theory decompositions. |
categories/B1-causal-direction.md | Category B1 โ causal-direction methods, with the reason they are adjacent rather than direct measurement. |
categories/B2-regime-stability.md | Category B2 โ regime-stability and distribution-shift measures. |
categories/C1-fdr-selection.md | Category C1 โ false-discovery-controlled feature-selection methods that consume orthogonality scores. |
categories/C2-finance-frameworks.md | Category C2 โ finance-specific applied frameworks such as hierarchical risk parity and clustered MDA. |
laymans/multi-symbol-and-broadening-explained-en.md | Plain-English explainer of the two follow-on tests (multi-symbol replication and panel-wide broadening). |
laymans/orthogonality-metrics-explained-en.md | Plain-English explainer of the whole catalogue: folk story, real story, what was built. |
verdict.md | Root verdict closing Phase 1: what is held, what is deferred, seven recorded methodology lessons, three next paths. |
findings/dashboard/build_audits.py from findings/evolution/audits/2026-05-16-orthogonality-measurement-instruments/AUDIT_LEDGER.json โ never hand-edited. Each quote was verified to occur in the file named beside it when the ledger was written.