SAVEPOINT DISK TIER — the chunk-streaming mover, measured (2026-09-08)
=====================================================================

Question (#2299 / #2302): a savepoint's life ends at a ring wrap because there
is no tier beneath the pinned ring.  What does it cost to put one there, and
what is the break-even against re-prefilling instead?

INSTRUMENT AND DISCIPLINE
-------------------------
Box 206, Corsair MP600 CORE XT (QLC) 3.6 TB, ext4 on LVM, /tmp on the root LV,
drive 96% full (159 GB free of 3.6 TB).  Everything below is produced by
`tools/savepoint/nvme_read_probe.py`, which drives the SHIPPED code path —
`arbi_serve.cache.chunk_mover.ChunkMover` over `LocalDirectBackend`, through
`savepoint_blob_io.write_savepoint_blob` / `read_savepoint_blob` — on a blob
the size of one savepoint for the hybrid this was designed against
(154,927,104 B = 147.75 MiB).  Run inside registry.arbi.work/arbi-serve
:test-4a2f15c.  No GPU: every number here is the host and drive halves only;
the H2D leg is not measured in this file.

  python tools/savepoint/nvme_read_probe.py --dir <dir> --repeats 3 --warm-dst

Two confounds this file avoids, and any later A/B must avoid the same two:

  * PROBE BLOBS ARE RANDOM, NEVER ZEROS.  A zero-filled 147.75 MiB file read
    back with O_DIRECT measured 21.1 ms / 7.35 GB/s on this drive; the same
    file written with real bytes measured 31.3 ms / 4.95 GB/s.  The first
    number is the FTL's trim path, not the flash.  `measure_disk_rates` fills
    its probe with `torch.randint`, and the sweeps do too.
  * READS ARE O_DIRECT, so a re-read is a genuine trip to the drive rather
    than a page-cache hit.  The drive's own write cache is still warm from the
    probe's write, which makes every read figure here the OPTIMISTIC end; a
    blob read minutes later reads at or below it.

THE HEADLINE, AND IT IS NOT THE CIPHER
--------------------------------------
A restore that allocates its own 147.75 MiB destination costs ~100 ms.  The
SAME restore into a REUSED destination costs 36.5 ms.  The 64 ms difference is
neither I/O nor crypto — it is the kernel faulting in a fresh anonymous
mapping on first touch, one minor fault per 4 KiB page, ~37,800 of them.

    chunk depth   fresh dst   reused dst   decrypt   drain   stall (reused)
       2 MiB  4    101.1 ms      40.7 ms    13.5      11.6     14.1
       4 MiB  4    105.1 ms      36.5 ms    12.5      11.7     10.9
       4 MiB  8    100.5 ms      37.4 ms    13.3      14.8      8.1
      16 MiB  4    103.4 ms      45.8 ms    11.9      12.9     20.2

So the single largest cost in a naive restore is an allocation, and it goes to
ZERO for a caller that hands in a pooled buffer.  `read_savepoint_blob(...,
dst=)` is that seam, and `ChunkBufferPool` pre-touches every buffer it hands
out for exactly this reason.  The destination arena for restored entries
belongs to the pinned-ring owner (#2302); until it exists, a restore pays the
fault storm and the receipt says so rather than quoting 36.5 ms.

At 36.5 ms the restore is READ-BOUND, which is the design claim and is now
checkable per stage rather than by argument: decrypt 12.5 ms, drain 11.7 ms,
stall-on-read 10.9 ms, against a raw O_DIRECT read floor of 31.3 ms.

THE SAME MISTAKE, MADE AND CORRECTED, IN THE CIPHER
---------------------------------------------------
An earlier pass in this session measured AES-256-GCM decrypt at 2.0 GB/s
(77 ms) and concluded the cipher, not the drive, was the binding leg.  That
was the same page-fault storm wearing a different hat: `update_into` writing
into a FRESH mapping.

    decrypt one 147.75 MiB blob, 4 MiB chunks, cryptography/OpenSSL, AES-NI
      into a fresh anonymous mmap                88.9 ms   1.74 GB/s
      into another fresh anonymous mmap          94.0 ms   1.65 GB/s
      into the SAME mapping, second pass         15.8 ms   9.81 GB/s
      into the SAME mapping, third pass          15.6 ms   9.93 GB/s

Through the shipped pipeline, with pooled buffers, the cipher measures
11.4-13.7 GB/s — 12-14 ms for a whole savepoint, and NOT the binding stage.
It is single-threaded on purpose: at ~11 GB/s it is memory-bandwidth bound, so
threads buy nothing.

THE PIPELINE'S TWO KNOBS
------------------------
Restore, into a fresh destination (so the drain column carries the fault cost;
the ordering of the rows is what the knob decides, not the absolute value):

    by chunk, at depth 8              by depth, at 4 MiB chunks
      1 MiB    95.2 ms  1.63 GB/s       2   146.8 ms  1.06 GB/s
      2 MiB   102.7     1.51            3   110.0     1.41
      4 MiB    96.4     1.61            4   108.0     1.43
      8 MiB    97.1     1.60            6   101.8     1.52
     16 MiB   118.4     1.31            8    99.0     1.56
     64 MiB   128.7     1.20

Shipped: CHUNK_BYTES = 4 MiB, PIPELINE_DEPTH = 8 (= 32 MiB of buffers per
direction).  4 MiB is flat with 1, 2 and 8 MiB and clearly better than 16 and
64; 64 MiB is the worst row in every sweep taken today.  Depth is monotone up
to 8 and its gain is entirely in the stall column (48.0 ms at depth 2 down to
2.3 ms at depth 8), which is the pipeline filling — the reader count is
depth-1.

RAW DRIVE CURVE, WITHOUT THE PIPELINE
-------------------------------------
O_DIRECT preads of a genuinely-populated 147.75 MiB file, no cipher, no drain:

      read size (QD 2)                 queue depth (16 MiB)
        1 MiB   51.0 ms  3.04 GB/s       1   35.1 ms  4.42 GB/s
        4 MiB   31.9     4.86            2   31.6     4.91
       16 MiB   31.7     4.89            4   31.5     4.92
       64 MiB   31.4     4.94            8   31.6     4.90
      128 MiB   31.3     4.95           16   31.6     4.91

NOTE A DISAGREEMENT WITH #2299's OWN TABLE, STATED RATHER THAN SMOOTHED.  That
table has 16 MiB peaking at 4.03 GB/s with 128 MiB the worst at 2.31, and QD 8+
collapsing.  Today this drive is flat from 4 MiB up and flat from QD 2 up.  The
QD finding reproduces at the knee (QD 1 is 11% worse); the read-size cliff does
not reproduce at all.  Same box, same drive, different fill state and a
different harness.  The constants shipped here are chosen on the PIPELINE
sweep, which is the path the store takes, not on either raw curve.

WRITE, AND FRAGMENTATION
------------------------
    spill one blob    68-101 ms   1.5-2.3 GB/s   (encrypt 13.7 GB/s)
    extents after     2, 13, 14, 15 across runs  (alarm at 64)

`fallocate` runs before the first byte and the 37 chunk records are appended
in order.  The USENIX FAST'24 cliff is at a degree-of-fragmentation of ~64;
this drive at 96% full cannot give a contiguous 148 MiB extent and hands back
2-15, which is an order of magnitude inside the cliff.  `extent_count()` reads
it back through FIEMAP — the same ioctl `filefrag` uses — after EVERY write, so
the day it stops being true is a warning in the log rather than a latency
nobody can explain months later.

The single-`pwrite` requirement in #2299 did not survive the chunked format and
should not: the blob is now 37 individually-authenticated records, so it is
one `fallocate` and 149 sequential appends.  The fragmentation argument is
satisfied by the preallocation, which is the part FAST'24 measures; what it
warns about is scattered sub-32 KiB writes, which 4 MiB appends are not.

FORMAT
------
    [ 4096 B header block: magic, version, nonce prefix, payload length,
      chunk size, chunk count, and the payload owner's opaque manifest ]
    [ chunk 0: AES-256-GCM ciphertext || 16 B tag, padded to 4096 ]
    [ chunk 1 ... ]

  * chunk record = 4,198,400 B for a 4 MiB chunk — 4080 B of padding, 0.097%.
  * nonce = 8 random bytes per blob || 4-byte big-endian chunk index, so
    nothing per-chunk is stored and a chunk replayed at another index opens
    under the wrong nonce.
  * AAD = the whole header block || (index, chunk count, chunk length), so an
    edited manifest or a reordered stream fails authentication rather than
    decoding into differently-shaped tensors.
  * verified in tests: a reordered blob raises InvalidTag, a truncated one is
    refused before a byte reaches the destination, an edited tenant id in the
    manifest fails, and a shredded key makes every blob unreadable.

THE BREAK-EVEN, AND THE ONE INPUT THAT IS STILL MISSING
-------------------------------------------------------
`SavepointTierBreakEven` derives the threshold from measured inputs only:

    cold_restore_us = slot_bytes / measured_restore_rate  +  ring_restore_us
    break_even_tokens = ceil(cold_restore_us / prefill_us_per_token), page-rounded

`measured_restore_rate` comes from `RecurrentSavepointStore.measure_disk_tier`
at boot (the numbers above); `ring_restore_us` from
`MultiStatePool.measure_savepoint_ring_transfer_us`, which the boot already
runs and already prints in the ring line.

`prefill_us_per_token` is NOT measured by this work and the tier writes NOTHING
until it is.  `tokens` returns None and the boot line says which measurement is
missing.  That is deliberate: an unvalidated threshold that ships is
indistinguishable from a measured one to every later reader, which is exactly
how `prefix_tier`'s DEFAULT_WARM/COLD_BREAK_EVEN_TOKENS became load-bearing
placeholders.  Two ways to supply it, both measurements:
`CacheConfig`-supplied from a run of `tools/short_prefill/sweep.py` (which
measured 0.188 ms/token asymptotic on this box for Qwen3.8-27B — see
docs/receipts/short-prefill-tax-2026-09-08.txt), or live from
`RecurrentSavepointStore.observe_prefill(tokens, elapsed_us)`, which keeps a
running mean and is the seam the engine should feed.

Worked example with that receipt's 188 us/token and today's numbers: a
147.75 MiB savepoint restores in 36.5 ms into a pooled destination, so the
break-even is ~195 tokens.  Every savepoint this store retains covers far more
than that, so the gate would admit all of them — which is the answer, but it is
NOT the shipped threshold, because 188 us/token belongs to one model on one
box and the store must not carry it as a constant.

WHAT IS NOT MEASURED HERE
-------------------------
  * the H2D leg and whether an O_DIRECT read and a GPU H2D contend at the
    memory controller (blocked on a free card).
  * the write rate under sustained load, past the QLC SLC cache.
  * per-layer restore overlapped with a forward (#2302): the fence is
    expressed per-layer (`LayerReadiness`) so that becomes a policy change,
    but no overlap is implemented or measured.
  * an object backend.  The `StorageBackend` protocol exists and
    `LocalDirectBackend` implements it; nothing else does.
