โ€บNavigation

iteration 03 ยท card 78 S4 wiring ยท batch-6 implementation ยท 2026-07-02

Card 78 โ€” wired into every write surface; the watchdog is green again S4 DONE

← campaign board ยท previous: iter 02 (the engine part)

In plain language โ€” what happened this iteration

The maths was done; today was plumbing โ€” the step that once went catastrophically wrong. Computing a number inside the engine is not enough: it must survive the journey into the database. That journey crosses about ten hand-offs โ€” the engine's in-memory record, two different columnar serializers (one for live streaming, one for the "dead letter" emergency spool that catches bars when the database is briefly unreachable), a Python dictionary bridge, the master column catalogue, the database blueprint, and the human-readable column documentation. Incident #556 happened because eight features were computed perfectly but five of those hand-offs still carried an old hand-written column list โ€” the values were silently dropped at the boundary and the columns deployed 0% populated.

Today every hand-off learned about the new column, and โ€” more importantly โ€” every watchdog that exists to catch a missed hand-off was exercised and is green:

โ€ข The schema contract watchdog (red on purpose since iteration 02) re-checked that four independent descriptions of the table โ€” the Rust record, the database blueprint, the feature catalogue, and the column documentation โ€” now list exactly the same columns. It flipped green.
โ€ข The exact-count tripwires were deliberately moved from 85โ†’86 and 90โ†’91. They're exact equalities, not minimums โ€” they fire on a missed addition AND an accidental removal, on both the Rust and Python sides.
โ€ข The round-trip rehearsal: Rust wrote a real 91-column file through the emergency-spool path and Python's data library read it back and verified every column name, order, and the new column's value โ€” proving the value survives an actual serializeโ†’diskโ†’read journey, not just a code review.

Why this feature's number is trustworthy end-to-end now: the reference answers (iter 01), the bit-exact engine (iter 02), and the persistence plumbing (today) are each guarded by their own failing-by- default test. What remains is certification: formally proving the feature never peeks at future data (causality), always gives the same answer on the same input (determinism), surviving a full quality gate, and an adversarial review โ€” then the pull request.

Grounded this iteration

S4 (wire) complete for bar_hoeffding_phi_squared_midreturn_duration โ€” all write surfaces derived from feature_manifest.toml (the SSoT): manifest entry (13th bar_close feature) โ†’ schema.sql + column_comments.py + constants.py โ†’ arrow_export.rs (schema/builder/append/finish + extended in-file round-trip test) โ†’ helpers.rs dict serializer โ†’ clickhouse_writer/row.rs (CORE_COLUMNS, struct, from_completed_bar) โ†’ dead_letter.rs. Tripwires 85โ†’86 and 90โ†’91 (+ Python mirror). S7/S8/S9 artifacts (schema DDL, contract, manifest) landed early as S4 side effects โ€” their gates are green.

stage4/16
schema contractGREEN
arrow suite27/27
row suite11/11
roundtrip9/9 ยท 91 cols
completeness13 == 13 == 13

Gates re-verified this iteration

GateWhat it provesResult
schema_contract_testschema.sql โ‰ก types.rs โ‰ก manifest โ‰ก column_commentsGREEN (was RED by design)
arrow_export_testsRecordBatch = exactly 86 columns; hoeffding value round-trips (0.03125)27/27
row.rs test_core_columns_countCORE_COLUMNS = exactly 91 (clickhouse-sink feature)11/11
completeness logicBAR_CLOSE_FEATURE_COLUMNS โ‰ก manifest group โ‰ก schema.sql (13 columns)GREEN
test_dead_letter_roundtrip.pyRust-written 91-col Parquet reads back in polars, names+order+nullability exact9/9
โ‰ค1e-9 oracle gate (re-run)kernel still bit-exact after wiringGREEN ร—2 symbols

Notes for operator review

#Note
1Streaming crate test binaries (gap_event_tests etc.) fail to compile without binance-integration โ€” pre-existing on main, unrelated to this PR (feature-gated gap module; the mise lanes pass the feature explicitly).
2Manifest range recorded as [0.0, 1.0001] to honestly cover the finite-m grid bias (1.0000631 at m=199 under perfect monotone dependence only).
3repair_direct_parquet.py derives columns from ALL_OPTIONAL_COLUMNS โ†’ auto-inherits the new column via constants.py (no hand-list).

Next fire picks up here

S5 causality + S6 determinism: register the two-substrate hoeffding in the bar_close_robustness_test harness (causality / determinism / scale / crash-bounds lanes โ€” needs a (close, dur) hook; existing lanes are close-only). Then S10 parity (locate/extend the hermetic ephemeral-ClickHouse round-trip โ€” production tables NEVER touched), S11 robustness witnesses, S12 ADR (docs/adr/2026-07-02-bar-hoeffding-phi-squared-midreturn-duration.md), S13 mise namespace hoeffding:* (oracle / oracle-fresh / test / test-full / check-full / doctor + .claude/commands/hoeffding/*.md wrappers, description text mirrored verbatim), S14 fmt+lint+test+deny, S15 Attacker/Defender challenge, S16 commit โ†’ stacked PR (base = main). Worktree: ~/eon/odb-worktrees/batch6-78-hoeffding, all work uncommitted, ledger is truth.