iteration 03 ยท card 78 S4 wiring ยท batch-6 implementation ยท 2026-07-02
Card 78 โ wired into every write surface; the watchdog is green again S4 DONE
← campaign board ยท previous: iter 02 (the engine part)
In plain language โ what happened this iteration
The maths was done; today was plumbing โ the step that once went catastrophically wrong. Computing a number inside the engine is not enough: it must survive the journey into the database. That journey crosses about ten hand-offs โ the engine's in-memory record, two different columnar serializers (one for live streaming, one for the "dead letter" emergency spool that catches bars when the database is briefly unreachable), a Python dictionary bridge, the master column catalogue, the database blueprint, and the human-readable column documentation. Incident #556 happened because eight features were computed perfectly but five of those hand-offs still carried an old hand-written column list โ the values were silently dropped at the boundary and the columns deployed 0% populated.
Today every hand-off learned about the new column, and โ more importantly โ every watchdog that exists to catch a missed hand-off was exercised and is green:
โข The schema contract watchdog (red on purpose since iteration 02) re-checked that four
independent descriptions of the table โ the Rust record, the database blueprint, the feature catalogue,
and the column documentation โ now list exactly the same columns. It flipped green.
โข The exact-count tripwires were deliberately moved from 85โ86 and 90โ91. They're exact equalities,
not minimums โ they fire on a missed addition AND an accidental removal, on both the Rust and Python sides.
โข The round-trip rehearsal: Rust wrote a real 91-column file through the emergency-spool path and
Python's data library read it back and verified every column name, order, and the new column's value โ
proving the value survives an actual serializeโdiskโread journey, not just a code review.
Why this feature's number is trustworthy end-to-end now: the reference answers (iter 01), the bit-exact engine (iter 02), and the persistence plumbing (today) are each guarded by their own failing-by- default test. What remains is certification: formally proving the feature never peeks at future data (causality), always gives the same answer on the same input (determinism), surviving a full quality gate, and an adversarial review โ then the pull request.
Grounded this iteration
S4 (wire) complete for bar_hoeffding_phi_squared_midreturn_duration โ all write surfaces
derived from feature_manifest.toml (the SSoT): manifest entry (13th bar_close feature) โ
schema.sql + column_comments.py + constants.py โ
arrow_export.rs (schema/builder/append/finish + extended in-file round-trip test) โ
helpers.rs dict serializer โ clickhouse_writer/row.rs (CORE_COLUMNS, struct,
from_completed_bar) โ dead_letter.rs. Tripwires 85โ86 and 90โ91 (+ Python mirror).
S7/S8/S9 artifacts (schema DDL, contract, manifest) landed early as S4 side effects โ their gates are green.
Gates re-verified this iteration
| Gate | What it proves | Result |
|---|---|---|
schema_contract_test | schema.sql โก types.rs โก manifest โก column_comments | GREEN (was RED by design) |
arrow_export_tests | RecordBatch = exactly 86 columns; hoeffding value round-trips (0.03125) | 27/27 |
row.rs test_core_columns_count | CORE_COLUMNS = exactly 91 (clickhouse-sink feature) | 11/11 |
| completeness logic | BAR_CLOSE_FEATURE_COLUMNS โก manifest group โก schema.sql (13 columns) | GREEN |
test_dead_letter_roundtrip.py | Rust-written 91-col Parquet reads back in polars, names+order+nullability exact | 9/9 |
| โค1e-9 oracle gate (re-run) | kernel still bit-exact after wiring | GREEN ร2 symbols |
Notes for operator review
| # | Note |
|---|---|
| 1 | Streaming crate test binaries (gap_event_tests etc.) fail to compile without binance-integration โ pre-existing on main, unrelated to this PR (feature-gated gap module; the mise lanes pass the feature explicitly). |
| 2 | Manifest range recorded as [0.0, 1.0001] to honestly cover the finite-m grid bias (1.0000631 at m=199 under perfect monotone dependence only). |
| 3 | repair_direct_parquet.py derives columns from ALL_OPTIONAL_COLUMNS โ auto-inherits the new column via constants.py (no hand-list). |
Next fire picks up here
S5 causality + S6 determinism: register the two-substrate hoeffding in the
bar_close_robustness_test harness (causality / determinism / scale / crash-bounds lanes โ
needs a (close, dur) hook; existing lanes are close-only). Then S10 parity (locate/extend the
hermetic ephemeral-ClickHouse round-trip โ production tables NEVER touched), S11 robustness
witnesses, S12 ADR (docs/adr/2026-07-02-bar-hoeffding-phi-squared-midreturn-duration.md),
S13 mise namespace hoeffding:* (oracle / oracle-fresh / test / test-full / check-full /
doctor + .claude/commands/hoeffding/*.md wrappers, description text mirrored verbatim),
S14 fmt+lint+test+deny, S15 Attacker/Defender challenge, S16 commit โ stacked PR
(base = main). Worktree: ~/eon/odb-worktrees/batch6-78-hoeffding, all work uncommitted, ledger is truth.