finding · findings/evolution/shared_data/orthogonality_probe.md

The orthogonality probe — does a new feature actually tell us anything new?

A small read-only tool that checks, on real market data, whether a candidate trading-signal feature is genuinely novel — or just a near-copy of something we already track. It does this across many symbols, sensitivity settings, and time windows, so a feature only earns a "novel" verdict if it holds up everywhere, not in one lucky stretch.

What is this?

This page is a plain-English guide to orthogonality_probe.py — a Python script that lives in the shared data layer. "Orthogonality" is just a fancy word for independence: two features are orthogonal if they don't move in lockstep, so each carries its own information. The probe measures that independence.

It is the Axis-1 engine of a three-axis system. Axis-1 answers one question only: is this candidate feature different enough from the ones we already have? (The other two axes — covered by sibling probes — check whether a feature is free of hand-tuned knobs, and whether it behaves the same no matter how it's implemented.)

What it replaced

• It replaced an older approach (fetch_multi_slice.py) that first copied data into intermediate files (Parquet, a columnar storage format) and only then analyzed it.
• That old fetcher was never actually run — its index file (manifest.json) is empty.
• The new probe skips the copy step entirely: it queries ClickHouse (the fast analytics database where all the bar data lives) directly, one slice at a time, live on the bigblack server. Nothing to copy, nothing to keep in sync.

How it works

The probe walks a grid of cells. One cell = one combination of (symbol, threshold, time-slice):

• Symbol — which crypto pair, e.g. BTCUSDT (Bitcoin) or ETHUSDT (Ethereum).
• Threshold — how big a price move it takes to "close" a bar, expressed in dbps (decimal basis points: 1 dbps = 0.001%, so 250 dbps = 0.25%). Lower threshold = more, smaller bars.
• Time-slice — a two-month window of history (e.g. one slice covers Jan–Feb of a given year).

For each cell, on the real bars in that window, it computes two things:

• A fast Spearman correlation matrix (|ρ|). Spearman ρ ("rho") is a rank-based correlation: it scores how strongly two features move together, from 0 (totally independent) to 1 (perfect lockstep), ignoring the exact magnitudes and just looking at order. The probe uses a shortcut — ranking the data and then taking an ordinary correlation — which is 40–200× faster than the standard Spearman routine but gives the same answer. A high |ρ| against an existing feature = redundant.
• An h_norm "mass-collapse" score per feature. This catches a different failure: a feature that looks varied but really dumps almost all its values into one bucket (collapsed, low-information). It's computed as the spread of values across five quantile buckets (quintile-entropy) divided by log 5, giving a 0-to-1 score.

Every candidate from the CANDIDATE_REGISTRY is computed per cell and dropped into the comparison matrix, so each candidate gets a regime-robust verdict across all slices — a judgment that holds across bull markets, bear markets, and quiet ranges — instead of a single-window snapshot that might just be luck. At the end, the probe writes a per-candidate cross-slice summary: pass-rate, median and worst (max) |ρ|, and which cell was the worst. That summary is the result you evaluate.

Which data it looks at

The list of symbols, thresholds, and windows isn't invented by the probe — it's locked in a separate single-source-of-truth document (pre-spec.md). The probe simply mirrors that scope in code; it never redefines it. Locking the scope up front (a "pre-registration") is what stops anyone quietly cherry-picking favorable windows after seeing the results.

• The grid starts from pre-spec.md v1's 24 windows, then programmatically keeps only windows from 2021 onward (MIN_YEAR = 2021).
• The three 2020 windows (slice22/23/24) are dropped: Bitcoin and Ethereum bars only begin 2021-01-01, so 2020 windows would be empty or ragged.
• That leaves 21 active slices (slice01…slice21), and the 6-symbol panel comes out rectangular (no missing corners).
• A couple of cells are also dropped for real-world data problems: LTCUSDT@100 has only 26 bars total (broken), and ETHUSDT@100 ends 2025-06-17, so slice01/02/03 show up empty and are skipped automatically at runtime.

The safety rails

Read-only — it can never change the data

SELECT-only via the local clickhouse-client. No INSERT/ALTER/DELETE/DROP/OPTIMIZE. Designed to run on bigblack. Single-core math (BLAS) by default, to stay polite on a shared host.

In plain terms: the probe can read the database but is structurally forbidden from writing, deleting, or reshaping anything. It also keeps a light footprint so it doesn't hog the shared machine. And it never stores raw bar values — only summary statistics — so nothing sensitive leaks into its output files.

Convergence floors — don't trust a number computed on too little data

A "floor" is a minimum sample size below which a statistic isn't trustworthy. The probe enforces two:

Key numbers

How to add a candidate, and run it

Adding a new candidate feature is one line in build_candidate_registry():

("bar_xyz", lambda x: my_fn(x)),   # x is a 1-D float window (df['close'], length 200) -> float

In words: you give it a name and a small function that takes a 200-long window of prices and returns a single number. The probe handles the rest.

Running it on bigblack (read-only):

# one-cell sanity check (default BTCUSDT@250/slice01)
python orthogonality_probe.py --smoke

# baseline matrix only — no antropy needed
python orthogonality_probe.py --smoke --no-candidates

# full grid, all candidates, all 21 slices
python orthogonality_probe.py

# restrict
python orthogonality_probe.py --symbols BTCUSDT,ETHUSDT --slices slice01,slice14

Use the spike virtual-environment (it has the antropy / sklearn libraries pre-installed): ~/venvs/orthogonal-features-phase2-spike/bin/python.

Outputs land in a local results/ folder (git-ignored, lives only on bigblack): per_cell_verdicts.jsonl plus candidate_robustness_summary.json. Again — only summary statistics, never raw bars.

What's deliberately left out (for now)

This probe handles only the pairwise question: "is candidate A redundant with feature B, one pair at a time?" The deeper conditional-dependence question — "does the candidate add anything given the whole rest of the panel together?" — is a known next step, called CODEC/FOCI (the top pick for that "Tier A2" job). It would plug in exactly where the Spearman matrix is built, inside run_one_cell(), but it is intentionally not wired in yet. That work is tracked in the audit's METRIC-EVALUATION-PROTOCOL document.