โ€บNavigation

iteration 01 ยท card 78 S1 oracle ยท batch-6 implementation ยท 2026-07-02

Card 78 โ€” Hoeffding ฮฆยฒ oracle grounded S1 DONE

← campaign board

In plain language โ€” what happened this iteration

The mission of this campaign is to add four new "measurements" (features) to every price bar the pipeline produces. Think of a bar as a snapshot of trading activity; each feature is a number that describes some aspect of it โ€” this batch adds numbers that are deliberately different from everything already measured (they passed two independence audits), so a machine-learning model gets genuinely new information.

This iteration started feature #1 of 4: card 78, "Hoeffding's ฮฆยฒ (phi-squared)". In plain terms it asks: over the last 200 bars, is there any relationship โ€” of any shape โ€” between how much the price moved and how long the bar took to form? Classic correlation only spots straight-line relationships; Hoeffding's ฮฆยฒ spots any pattern (curved, U-shaped, threshold-like). It answers on a 0-to-1 scale: 0 = "move size and bar duration are unrelated", 1 = "one completely determines the other".

What was actually built today is the answer key, not yet the engine part. Before any feature is allowed into the production Rust engine, we first write a slow-but-trusted reference version in Python and freeze its answers on 10,000 real Bitcoin bars and 10,000 real Ethereum bars ("the oracle"). Later stages must reproduce these answers to within 0.000000001 โ€” that's how we prove the fast production code computes exactly the same mathematics as the reference.

Two safeguards worth explaining: (1) the reference computed every answer twice, via two completely different mathematical routes โ€” a grid-counting method and an exact whole-number formula โ€” and required both routes to agree on all 19,602 windows before writing anything (they did, to better than one part in a trillion). (2) The price inputs were passed through the same rounding step the production engine uses, so the reference and the engine literally see identical numbers โ€” a past incident (I3) taught us that even a rounding difference in the 9th decimal can flip a rank and corrupt this kind of statistic.

One surprise: the shared test dataset used by earlier features stores only closing prices, but this feature also needs each bar's duration. When we went back to the production database to fetch durations for the same bars, we found the database had been legitimately repaired since May (gap-healing), so those exact old bars no longer exist. Rather than corrupt the old dataset, card 78 got its own fresh, self-contained dataset (price + duration), fingerprinted (SHA-256) so any future tampering or drift is detected instantly.

Grounded this iteration

S1 (oracle) for bar_hoeffding_phi_squared_midreturn_duration: authored tests/fixtures/regenerate_bar_close_dur_samples.sh (read-only ClickHouse SELECT, pinned spec) โ†’ committed-ready 2-col fixtures (close, duration_us) for BTC+ETH; authored scripts/gen_hoeffding_oracle.py (clean-room numpy; I2 ratio-rank substrate; I3 FixedPoint pre-quantization drift guard, drift=0; I4 fail-closed dual-leg: sweep-kernel grid construction vs exact-integer closed-form, tolerance 1e-12, green on all 19,602 windows) โ†’ oracle CSVs for BTC+ETH, 9,801 populated rows each, all values finite and inside [0,1].

featurecard 78
stage1/16 oracle
stack78โ†’59โ†’60โ†’50
dual-leg xcheck19,602/19,602
quantization drift0
ฮฆยฒ range BTC0.005โ€“0.099
ฮฆยฒ range ETH0.004โ€“0.119

Artifacts + integrity pins

Artifactsha256 (prefix)Note
BTCUSDT-bars-close-dur-sample-10k.csv26bfea6f887ea114โ€ฆ10,000 real bars, close+duration_us, 250 dbps from 2024-01-01
ETHUSDT-bars-close-dur-sample-10k.csv1627ec312e502354โ€ฆโ‰ฅ2-symbol gate (scale-bug detector)
BTCUSDT-hoeffding-oracle-sample-10k.csv2ec3992da85820ccโ€ฆ9,801 rows, mean ฮฆยฒ=0.0281
ETHUSDT-hoeffding-oracle-sample-10k.csv686761231929d20aโ€ฆ9,801 rows, mean ฮฆยฒ=0.0213

Decisions made (for operator review at PR time)

#DecisionWhy
1New self-contained 2-col fixtures; close-only fixtures untouchedProduction cache repaired since 2026-05-29 (45 scattered diff hunks vs committed closes) โ€” old bar set irreproducible; existing oracle SHAs must not churn
2Returns computed WITHIN the 200-close window (m=199 pairs), pair = (close[t+1]/close[t], dur[t+1])Bartels precedent; probe convention pairs the return landing in bar i with bar i's duration
3Rank the IEEE-exact gross-return ratio, not lnI2 guard โ€” rank-identical to logret, immune to libm-vs-glibc near-tie rank flips
4I4 second leg = exact-integer closed form (no R)Canonical FOSS ref (copBasic::hoefCOP) is GPL-2 + R-side; registry records PORT / foss_oracle_validated=false; same convention as card 59's closed-form leg
5Whole-window NaN policy (not probe's row-dropping)bar_close family kernel contract; wiring maps NaNโ†’None/NULL

Next fire picks up here

S2/S3 (xcheck + kernel): add compute_bar_hoeffding_phi_squared_midreturn_duration(closes, durations_us) to bar_close_features.rs (first two-substrate kernel โ€” needs a durations window alongside recent_bar_closes in the processor), ordinal stable ranks, property battery + in-Rust independent leg + โ‰ค1e-9 oracle parity over BTC+ETH. Then S4 wire: 13th walk-tuple column (all 12 existing destructures gain a field), push_and_compute_bar_close gains durations, manifest entry, and the #556 write surfaces. Worktree: ~/eon/odb-worktrees/batch6-78-hoeffding (branch feat/bar-orth/batch6-78-hoeffding off origin/main @ 4c0ee656). All work uncommitted on disk โ€” ledger is the state truth.