opendeviationbar · feature-building · plain-English explainer

How These Features Are Built PLAIN-ENGLISH

A walkthrough, in my own words, of what I have been doing: how I find a candidate feature, how I decide it is worth keeping (genuinely new, not a re-labelled copy of something we already have), how I check that judgement holds up as the market changes, and how I turn a survivor into a real production column I can trust. Every claim links to the page that grounds it.

1 · What I am actually trying to do

The strategy already has a large panel of feature columns. Adding another column is only worth it if it tells us something the panel does not already tell us. So the whole job is a filter: take a long list of candidate features, and keep only the ones that are (a) orthogonal — genuinely new information, (b) parameter-free — no hand-tuned magic numbers, and (c) agnostic — one well-defined algorithm two honest people would implement the same way. Those three tests are the Three-Axis Probes, and one gate combines them into a single keep / skip decision.

2 · How I decide a candidate is orthogonal — the metrics, divided

I never trust a feature because its paper says it is novel — I measure its redundancy on our own bars. The metrics fall into three buckets: the operational ones that actually decide the verdict today, a complementary second check that is built but not yet wired into the decision, and the ones that are only reported, never gating.

① Operational — how a candidate is actually screened OPERATIONAL

The real in-production screen: "is this candidate new information vs the columns we already ship?", scored on real bars, worst-case across market regimes. Important: crypto and forex were screened by two different probes — so I show them separately, exactly as they ran.

② Complementary second check — built, not yet operational NOT OPERATIONAL

A different axis: instead of "new vs production", it asks "is this candidate redundant with the other candidates in the batch?", to weed out near-duplicates among the new features. Terry classes this as a 2nd-layer / complementary check — explicitly not a meta-evaluation. It lives only inside the rotation probe (rotating_slice_orthogonality_probe.py), which is not wired into the operational decision.

How the rotation probe scores (also not operational): each candidate is tested across many strata — its grid is asset class × symbol × threshold × volatility × session × trend, run across both forex and crypto — and only a candidate that survives the hardest stratum and repeats in both markets is kept. Here "asset class" = crypto vs forex, and it is a dimension only in this probe — the operational probe above is crypto-only. Detector sanity check: a shipped feature scored against itself returns ρ ≈ ξ ≈ LOO-R² ≈ 1.0 — it correctly flags a perfect duplicate.

③ Other metrics — measured and reported, but NOT gating

References: the operational decision is the orthogonal dashboard (gating metrics + the report-only distribution view); the complementary LOO-R² check is the rotation probe; see also slicing methodology and the named regime slices.

3 · How I build a feature once it is chosen

I do not invent a new engineering path each time. I review the previous implementations and follow the same robustness standard, building the new feature step by step in an autonomous loop. The loop walks a feature from "trusted formula on paper" toward a live, backfilled production column — but it never merges, deploys, or backfills by itself: it stops at an open PR, and a human takes it the rest of the way.

The full lifecycle — chosen candidate → live, backfilled column

Stages ①–③ are the autonomous loop (detailed in §4). Stages ④–⑦ are operator-gated: the human merges, deploys with mise run deploy:bigblack (builds + installs under the odb service account), restarts the sidecar so new live bars carry the column, then runs the historical backfill so it also fills back across history — consistent at the live↔historical boundary. (Forex is the same shape from /opt/mql5 → fxview-sidecar.)

Before I merge — closing the loop out

Once the loop reaches its open PR, I close it out in four steps (these are the loop's own P2 / P3 phases plus my operator read):

1. Review the surfaced findings — read everything the loop flagged.
2. Advisory audit — a pass dedicated to closing any identified gaps.
3. Attacker / defender review session — adjudicate the gaps the two adversarial agents surface.
4. Further advisory audits — confirm everything is aligned and genuinely mergeable.

4 · Loop-in phases — crypto and forex

Crypto and forex run as two separate autonomous campaigns, each with its own pipeline. They share the same discipline (bit-exact oracle → causal Rust kernel → full wiring → adversarial challenge) but differ in shape: crypto runs a 16-stage-per-feature pipeline across four phases; forex runs a leaner per-feature lane with stacked PRs.

🪙 Crypto — loop-in phases (latest: batch 5 · PR #544 · 8 features)

Four campaign phases. Inside P1, each feature walks all 16 stages before the next begins; then the whole batch is verified and adversarially challenged.

The 16 per-feature stages inside P1 (in order):

🪙 Forex — loop-in phases (latest: RQW + CREx · stacked PRs #124 → #126 → #127)

Forex runs one lane per feature, each stacked on the previous PR; schema and code ship together in a single PR (no split-brain).

Both flows then enter the operator-gated tail in §3 (merge → deploy → sidecar restart → backfill). Source campaigns are the newest in the dashboard: crypto 2026-06-19-bar-orth-batch5-implementation (PR #544) and forex 2026-06-24-forex-batch34-rqw-crex-implementation (PRs #124 / #126 / #127). Browse from the crypto and forex candidate hubs.

🗺 The whole pipeline at a glance — discovery → ClickHouse column

The current, operational path a crypto feature actually travels, end to end. Green = running today; amber = a human must trigger it (the loop stops at the PR and never auto-deploys); grey = not in this chain.

5 · Why I trust the result

• Bit-exactness (oracle + parity stages) means production runs the exact maths I validated — not a re-interpretation of it.
• Worst-regime screening across all 7 regime epochs means a survivor is not a one-window fluke; it had to stay independent in the worst regime cell, not just on average.
• Causality + determinism stages prove a feature only ever reads past bars and returns the same answer every run.
• Adversarial challenge + one human-merged PR per feature means every claim is attacked (Attacker/Defender) before it is trusted, and a person — not the loop — decides to merge, deploy, and backfill.

6 · What I have been doing

Beyond implementing individual features, recent work has been about strengthening the orthogonality judgement and looking ahead — a forward-looking probe, a complementary cross-check, and a hardened candidate registry. Important: these are research / evaluation surfaces — only the production decision in §2 is operational today; the three below are not yet wired into the live gate. Each is grounded in a page you can open:

Reference note: none of these three is wired into the operational gate yet — the forward probe and rotation probe are EVALUATING, and the candidate registry is a hardening campaign over a tracking ledger. Only the production decision in §2 is operational today.

♻️ Rotation Orthogonal Probe — full workflow NOT OPERATIONAL

The complementary cross-check, step by step (built, not wired into the live gate). The cyan callouts flag where the crypto and forex orthogonality probes genuinely differ.

🗃 Registry-based candidate discovery — full workflow NOT OPERATIONAL

How a candidate would flow from an idea to a live column through the hardened registry. The key idea: a cheap novelty pre-filter (is it already in our catalogue?) is a different gate from the expensive orthogonality test (does it add new information?).

7 · Candidates contributed so far — and how each was promoted

Full transparency on what shipped and the exact bar each one cleared. There is no single promotion standard — three were used, and they differ by group:

• Crypto batch-5 (8 features) — the Median Rule on the literal orthogonality_probe.py: median|ρ| ≤ 0.85 ∧ pooled-pass ≥ 60% ∧ min-regime-pass ≥ 50% (Spearman |ρ| + h_norm; ξ not computed).
• Crypto bar_close (4 features) — an older, stricter predecessor harness that additionally gated on R² ≤ 0.85 (worst-valid-cell, ~10 slices).
• Forex (all 13 features) — a two-stage probe ending in the Chatterjee ξ keeper gate: ξ > 0.50 per-symbol-median + ≥2-symbol replication.

🪙 Crypto — 12 contributed columns

💱 Forex — 13 contributed columns

Honest notes: (1) the FD pair (bar_petrosian_fd / bar_katz_fd) were promoted before orthogonality_probe.py existed — their "re-confirmed by the probe" rests on antropy determinism (recompute == materialized is asserted, not measured). (2) On the counts: the batch-1/2 ξ stage produced 14 ξ-confirmed keepers (13 KEEP + mann_kendall_z as the direction-clique representative), of which only 9 became live columns (5 deferred — substrate / parameter-blocked); batch 3-4 ξ-cleared 15, of which 4 shipped. The 13 contributed columns on this page are those 9 + 4, and mann_kendall_z is one of the 9 (column #6) — not an extra on top of the 13. Full per-column reference: forex feature catalog.

8 · Where everything connects