SAVEPOINT DISK TIER — the seams wired, and the first boot that served off disk
==============================================================================

Scope. This receipt covers the WIRING of the tier (arbicity/arbi-serve#2312):
the three seams #2299 left stubbed, plus four defects that only a booted
engine could show. It is a smoke reading taken by the author of the wiring,
not the measurement of record: there is no paired tier-off control here, and
the arms below differ in concurrency, so nothing in this file is a claim
about what the tier is WORTH. The A/B belongs to a separate run.

BOX AND BUILD
-------------
Box 206, GPU 1 (RTX 4090, 450 W, no clock pins). GPU 0 left to a peer.
Host: 93 GB RAM, earlyoom inactive. Drive: Corsair MP600 CORE XT (QLC)
3.6 TB, ext4 on LVM, 96% full (148 G free) — a state that matters, see the
write rate below.

Image registry.arbi.work/arbi-serve:test-4a2f15c with arbi_serve bind-mounted
from feat/savepoint-tier @ 5207240c0. Qwen3.8-27B-exl3-4.0bpw,
--kv-cache-dtype=tkv, TKV_BITS=4, vq2_hadamard_ladder234_mtp_20260827,
--max-batch=8 --chunk-prefill=2048 --max-batched-tokens=2304.
EOS HONORED (no --ignore-eos): with it a thinking model never closes its
think block, the turn renders no answer, and every retention arm is
flattered.

  ARBI_SAVEPOINT_DISK_DIR=/cache/savepoint-tier-e2e
  ARBI_SAVEPOINT_DISK_GB=6
  ARBI_SAVEPOINT_RING_SLOTS=32

  docker compose --env-file serve-local.env \
    -f compose.serve.yaml -f compose.bindmount.yaml -p tier-e2e up -d
  (ARBI_IMAGE, ARBI_SRC_ROOT, ARBI_GPU_DEVICE=1, ARBI_PORT=8021,
   ARBI_ENV_FILE=<the three vars above>)

  python3 tools/savepoint/retention_scenarios.py http://127.0.0.1:8021 \
    --scenarios linear --sessions 22 --turns 4 --filler-reps 120 \
    --max-tokens 300 --think-time-s 12 --replay-reasoning --nonce back1

WHAT THE BOOT MEASURES, AND WHAT IT REFUSES TO ASSUME
-----------------------------------------------------
  savepoint ring: 32 slot(s) x 154927104 B = 4957667328 B PINNED HOST
    — measured this boot: write 11748 us, restore 11597 us per slot
  savepoint disk tier: HOLDS NOTHING — a cold restore of 154927104 B
    measures 89.6 ms on this drive (read 1.99 GB/s, decrypt 14.07 GB/s,
    host-to-device 11.6 ms), but the prefill rate it has to beat has not
    been measured this boot.
  savepoint flush band: started against a measured 2.53 GB/s write, one
    entry (154927104 B) of burst.

The tier declines to write until the third input exists. It arrives from the
served path: the scheduler brackets a step from schedule() to commit_state,
and the break-even appears once observations land —

  break_even_tokens=512   prefill_observations=12

THE RUN — 22 conversations, 4 turns, 12 s think-time
-----------------------------------------------------
  requests 88, errors 0, reuse turns 66
  cached% mean 89.9, median 93.0, misses 2
  TTFT median 0.192 s over reuse turns (16.277 s at turn 0, a cold prefill
    of a ~6 k-token prompt under 22-way concurrency)
  decode 47.4 tok/s median

Counters, verbatim from the engine's own line:

  evictions_that_waited_for_a_flush=0     <- THE invariant. Zero.
  flushes_from_band=74                    (writes that LANDED)
  flushes_lost_the_race=870               (entry gone before the writer)
  flushes_deferred_no_budget=1721         (the token bucket, working)
  flushes_yielded_to_restore=15
  store.flush_total=74
  store.flush_bytes_written=11464605696
  store.evict_clean_free_list=12          (evictions that cost nothing)
  store.restores_from_disk=13             <- the tier SERVED
  restores_from_disk_seen=13              (admission side agrees)
  store.spill_gate_reject=106
  store.max_blob_extents=12
  resume_total=98
  arena.warm=0  arena.cold=13
  break_even_tokens=512  prefill_observations=12

Per stage, over the 13 restores that actually happened:

  restore        97.2 ms
  decrypt        12.6 ms
  drain           8.2 ms
  stall-on-read  67.6 ms

READ-BOUND, which is the design claim, now measurable per stage on a served
restore rather than argued. And 97.2 ms against the fresh-destination figure
in savepoint-disk-tier-2026-09-08.txt is the arena's own null control:
arena.cold=13 / arena.warm=0 says every one of these allocated its
destination, so the reuse win is still entirely on the table.

Disk budget: 42 blobs, 6.0 G, against ARBI_SAVEPOINT_DISK_GB=6. The LRU
unlink enforces the byte budget; the root filesystem did not move (96%).

WHAT THIS RUN DOES NOT SHOW
---------------------------
* No paired tier-off control. The two arms in this file differ in
  concurrency (12 vs 22 sessions), so the decode figures are not comparable
  and NOTHING here says what the tier costs or saves.
* arena.warm=0. Every restored entry stayed resident, so no region was ever
  returned and reused. The pooling is exercised in the CPU suite; on a card
  it is unproven.
* decode_baseline_n=0. Every prefill observation came from a step with no
  decode rows; the marginal path (mixed step minus a decode baseline) did
  not fire here, because the decode steps commit through the MTP route and
  do not reach this seam. steps_stamped=2553 vs steps_closed=12 is that gap.
* Losslessness is demonstrated in the CPU suite on the shipped path
  (test_a_restored_savepoint_is_bit_identical_to_the_one_that_was_stored),
  byte for byte on fp32 tensors through the drive and AES-256-GCM. It is not
  separately demonstrated on-card here.

FOUR DEFECTS THIS BOOT FOUND THAT NO CPU TEST COULD
----------------------------------------------------
1. THE BLOB HEADER DID NOT FIT. A savepoint's manifest carries one
   descriptor per slab; this model has 96, serializing to 4825 B against a
   header that assumed a single 4096 B block. EVERY served write was
   refused, the tier directory stayed empty, and no counter said so. The
   header is now an integral number of DIO_ALIGN blocks with the count in
   the header itself.

   The reason no probe caught it: measure_disk_rates built its probe from
   ONE tensor, so the boot probe — and nvme_read_probe.py, and therefore the
   whole savepoint-disk-tier receipt — measured a blob whose header is
   structurally incapable of reproducing the failure. The probe now builds
   at the pool's own slab count. That change, not the header fix, is what
   stops this recurring.

2. EIGHT FLUSH BANDS. build_active runs several times per boot; each pass
   built a store, a ring and a scheduler, and started a flusher. Seven
   daemon threads held stores nothing served from, and the savepoint tier
   line an operator reads came from a dead one — reporting zero steps and
   zero restores while the live tier was serving resumes.

3. THE PREFILL INTERVAL NEVER CLOSED. The scheduler emits a list;
   StepPlan.from_slate carries a tuple built from it; the commit side sees
   the second. Matching on object identity closed nothing: 216 opened, 0
   closed, which reads exactly like an engine with nothing to measure.

4. A DEMOTED ENTRY WAS UNREACHABLE. _pop_locked discarded the match index,
   so best_match could never offer a key whose bytes were on disk — a disk
   restore was reachable only through the match/get race. The index now
   tracks reachability: resident OR on disk.

   And the flush band reported 57 flushes while writing zero bytes: it
   counted attempts, offered keys with no resident bytes left, and skipped
   every trail past the window as "clean by construction" — which is a goal
   the band exists to reach, not a fact. Under a ring there is no second
   chance: a wrap takes the bytes for another request and the eviction-side
   flush never sees the entry.

A NOTE ON THE WRITE RATE
------------------------
The band's budget is whatever the boot's probe measured, and on this drive
at 96% full that has been seen anywhere from 0.02 GB/s to 2.53 GB/s across
boots. The low readings were probes running concurrently with the weight
load; only the last build_active's store now serves, and its probe runs
after the weights are in. flushes_deferred_no_budget being large is the
budget doing its job — the band has more work than the drive will take — not
a fault. It is worth pinning down separately whether the probe is timing the
transfer or the create/fsync on a full QLC drive.
