SAVEPOINT RETENTION — the middle tier: pinned STAGING, pageable RETENTION
=========================================================================

WORKLOAD, STATED FIRST BECAUSE IT BOUNDS EVERYTHING BELOW
---------------------------------------------------------
Prompts here are SHORT. `--filler-reps 120` produces a prompt of **5,355 to
6,343 tokens** (measured per arm, not assumed), and every turn resends the
transcript. arbi-serve is a long-context engine whose canonical measurement
shape is 128k, so **this is not the measurement of record** — a long-context
MIX reported per length band is, and it is pending on a separate harness.

Read every number below with this in front of it: **a savepoint weighs the
model's recurrent geometry, 154,927,104 B across 96 slabs, whatever it
covers.** The boot line says so itself:

  recurrent state: 96 slabs [one (layer, attr) tensor each, 9 rows =
    max_num_seqs + 1]; one row = 154927104 B [one sequence's state across all
    96 slabs; what a savepoint ring slot holds; invariant in max_num_seqs]

So the COST of retaining one is invariant in the prompt while its VALUE — the
prefill it avoids — scales with the tokens it covers. At ~6,300 tokens a
retained savepoint stands in for ~1.2 s of prefill; at 128k the same fixed
148 MiB stands in for roughly twenty times that. **This workload is close to
the worst case for a retention tier**, and nothing here is tuned against it:
the sizing below follows a measured signal precisely so that no number in this
file has to be right at 128k for the mechanism to be.

QUESTION
--------
`docs/receipts/savepoint-tier-ab-2026-09-08.txt` established that a ring of 64
slots fixes the hit rate by holding the working set and costs 9.92 GB of
PAGE-LOCKED host RAM, while the disk tier recovers the same hit rate at ring
32 by writing ~11.9 GB to NVMe while serving and paying a 37% decode
regression for it. Both are corrections for ONE defect: staging and retention
were the same buffer, so retention had to be page-locked and had to be sized
by `max_num_seqs` rather than by the number of conversations in flight.

This file asks the only question that decides whether splitting them is worth
doing: **what is the hit rate, and what does the memory cost — page-locked and
total?** TTFT and decode are the do-no-harm check, not the result.

INSTRUMENT
----------
Box 206 GPU 1 (RTX 4090, 450 W, no clock pins), GPU 0 idle. 93 GB RAM,
earlyoom inactive. Image registry.arbi.work/arbi-serve:test-latest, arbi_serve
bind-mounted per arm. Qwen3.8-27B-exl3-4.0bpw, --kv-cache-dtype=tkv,
TKV_BITS=4, bundle qwen3.8-27b-exl3-4.0bpw_k4v4_mtp_20260816, --max-batch=8
--chunk-prefill=2048 --max-batched-tokens=2304. EOS HONORED.

`savepoint_disk_dir` EMPTY IN EVERY ARM. The disk tier is inert throughout —
no arm wrote a byte to a drive — and each boot states the regime before a
token is served: `savepoint disk tier: OFF — no offload directory`.

One container recreated per arm; each drove the identical command under its
own nonce, so no arm reads a cache another filled:

  tools/savepoint/retention_scenarios.py --scenarios linear --sessions 22
    --turns 4 --filler-reps 120 --max-tokens 300 --think-time-s 12
    --replay-reasoning

  B   main @26744d3d0   RING_SLOTS=32   the shipped default
  D   this branch       ring AUTO       staging-only ring, pool at min_entries
  C   main @26744d3d0   RING_SLOTS=64   the pinned-RAM answer
  D2  this branch       ring AUTO       + retention sized by measured demand

D and D2 differ in ONE thing: whether the pageable pool is bounded by the
configured `savepoint_min_entries` or by the observed conversation working
set. D is kept in the table because the difference between the two rows is the
whole argument for demand-following.

A DEFECT THIS RUN FOUND, AND IT IS NOT ABOUT SAVEPOINTS
--------------------------------------------------------
**The pinned-host ledger books what a reservation ASKED for, and the ask is
under what it COSTS by up to a factor of two.** That ledger exists for exactly
one reason — page-locked pages are neither swappable nor reclaimable, so an
over-reservation does not degrade, it ends in the kernel's global OOM killer
choosing a process — so under-pricing it is precisely the failure it was built
to prevent. On the arm-C boot every later page-locked reservation was priced
against a host believed to have ~7 GiB more than it did.

It is in the shipped default and it has nothing to do with retention. It was
found while chasing an unexplained RSS column (below), which is the only
reason it appears in this file rather than its own.

THE MEASUREMENT. On 206 GPU 0, no engine in the process, allocating through
the ring's own call — `torch.empty(n, uint8, cpu, pin_memory=True)` — reading
VmRSS either side, with ORDINARY PAGEABLE memory of exactly the same size
measured the same way as a CONTROL so the probe could fail:

  CONTROL pageable, touched   asked 2.309 GiB   resident +2.310   ratio 1.000
  PINNED  (the ring's call)   asked 2.309 GiB   resident +4.000   ratio 1.733
  PINNED  again, same size    asked 2.309 GiB   resident +0.000   ratio 0.000

The control costs exactly what it asks. The pinned request costs 1.733x, and a
SECOND request of the same size costs NOTHING. That pair is the mechanism: not
pinned pages being expensive per page, but a CACHING host allocator serving a
request out of a rounded block. Walked across the sizes the ring asks for, the
rule is exact — round UP TO THE NEXT POWER OF TWO:

   slots   asked GiB   resident GiB   ratio
       1       0.144          0.251   1.740
       8       1.154          2.001   1.734
      16       2.309          4.001   1.733
      32       4.617          8.001   1.733
      64       9.234         16.001   1.733

THE FIX. `acquire_shared_ring` now measures VmRSS across the allocation and
books what it COST, floored at the ask, naming both figures in a warning.
Measured rather than modelled, deliberately: the rounding rule is an
implementation detail of a dependency, a constant restated in this tree goes
stale silently, and a measurement survives the allocator changing. An
unreadable VmRSS leaves the booking alone — a sentinel, never a zero — and the
floor is the conservative direction, since reserving more than was taken costs
slots while reserving less costs the process.

WHERE THE CLIFFS ARE, for whoever sizes a ring next. At 0.144287 GiB a slot,
the block boundaries fall at 2, 4, 7, 14, 28 and 56 slots:

   slots   asked GiB   block GiB
      16      2.3086        4.00
      27      3.8958        4.00
      28      4.0400        8.00   <- cliff
      32      4.6172        8.00
      33      4.7615        8.00   (the SAME block as 32)
      55      7.9358        8.00
      56      8.0801       16.00   <- cliff
      64      9.2344       16.00

So the shipped AUTO of 16 sits well inside the 4 GiB block and staging could
grow to 27 slots for zero additional page-locked residency, while 28 doubles
the bill. That is stated so a reader can see the shape, NOT tuned against: the
ring's size is derived from staging (`max_num_seqs` times the hand-outs a turn
holds) and a slot count chosen to land under an allocator's block boundary
would be a constant borrowed from a dependency. The ledger now tells the truth
about the cost either way, which is the part that matters.

THE HEADLINE — memory at a given hit rate
------------------------------------------
PINNED is the ledger's BOOKED figure — what an operator reads at boot — and
RESIDENT is what those slots actually hold, from the probe two sections up.
Both are given because the two differ by the allocator's rounding, and the
booked one is the number every other reservation this boot was priced against.

                    cached%   PINNED  (RESIDENT)  PAGEABLE   total   engine RSS
  B  ring 32          20.8   4.62 GiB  ( 8.00)      0       4.62 GiB    —
  C  ring 64          87.7   9.24 GiB  (16.00)      0       9.24 GiB   21.0 GB
  D  pool@64          89.4   2.31 GiB  ( 4.00)   9.24 GiB  11.55 GiB   18.1 GB
  D2 pool@demand      89.3   2.31 GiB  ( 4.00)   4.47 GiB   6.78 GiB   13.1 GB

`D2` reaches `C`'s hit rate at **a quarter of C's page-locked host RAM** —
booked 2.31 against 9.24 GiB, RESIDENT 4.00 against 16.00 — and at **less
total savepoint memory than C**, 6.78 GiB against 9.24 GiB booked, 8.47
against 16.00 resident. Both halves are needed and `D` is why: splitting
staging from retention without sizing retention to demand converts page-locked
RAM into a larger amount of pageable RAM, which is a weaker claim.

THE RSS COLUMN, EXPLAINED
--------------------------
Savepoint accounting predicts C − D2 = 9.24 − 6.78 = 2.46 GiB. The measured
process RSS difference is 7.9 GB, ~5.4 GB more. That gap was reported here as
UNEXPLAINED and not claimed, and then probed; the probe is the section above.

The allocator's rounding closes it arithmetically. Real page-locked residency
is B 8.001 / C 16.001 / D2 4.001 GiB, so C − D2 residency is 12.000 GiB, less
D2's 4.47 GiB of pageable pool = 7.53 GiB = 8.09 GB predicted against 7.9 GB
measured. The cleaner check is at the ledger's own instant: subtracting
MEASURED residency instead of the booked figure leaves an engine working set
of 19.911 − 16.001 = 3.910 GiB in C against 7.912 − 4.001 = 3.911 GiB in D2 —
identical to a thousandth of a GiB, which is what "the same engine, a
different ring" has to read.

And it makes the split worth MORE than nominal: page-locked residency falls
from 16.001 GiB to 4.001 GiB, a 12 GiB reduction and a true factor of four
rather than the nominal 9.234 → 2.309.

The headline table's PINNED column is the ledger's booked figure, which is
what an operator sees at boot. Read the residency figures beside it.

The difference in KIND is the point of the first column. Page-locked pages can
be neither swapped nor reclaimed, so over-subscribing them ends in the
kernel's global OOM killer choosing a process; the pageable pool degrades into
a page-out and a fault storm and nothing else. The pinned figure is the
engine's own ledger row, not an inference:

  D2: HOST PINNED LEDGER … reserved 2.309 GiB | headroom 77.418 GiB
      (MemAvailable) | engine working-set floor 5.603 GiB (VmRSS − reserved,
      MEASURED)
        cache.savepoint_ring   2364.0 MiB  (16 slot(s) x 154927104 B)

  C:  HOST PINNED LEDGER … reserved 9.234 GiB
        cache.savepoint_ring   9456.0 MiB  (64 slot(s) x 154927104 B)

THE DO-NO-HARM CHECK — reuse turns only (t>0), n=66 per arm, 0 errors
----------------------------------------------------------------------
                          cached%   TTFT med    decode tok/s   uncached/turn
  B  main, ring 32          20.8     12.192 s      25.0           5689
  C  main, ring 64          87.7      0.270 s      49.6            450
  D  branch, pool@64        89.4      0.272 s      48.7            438
  D2 branch, pool@demand    89.3      0.181 s      49.0            450

Decode is flat across C, D and D2 (49.6 / 48.7 / 49.0), which is the number
that mattered: the copy-out is a host-to-host memcpy on a background pass that
QUERIES the snapshot fence rather than blocking on it — `cudaEventSynchronize`
spins, and a spinning background thread beside the engine's launch thread is a
decode debit paid to buy a TTFT.

WHAT THE MECHANISM REPORTS ABOUT ITSELF (D2, its own line)
-----------------------------------------------------------
  savepoint retention: pageable 31 region(s) of 31 bound = 4802867200 B held,
    peak 31 concurrent (THE conversation working set), warm=96 cold=31
    refused=0 trimmed=0 | promoted=127 bytes=19675742208 below_floor=0
    lost_the_race=0 no_region=0 | resident=31 (reachable=20 dominated=11
    working_set=14) bytes=4802740224/4802740224 demand_entries=16
    reuse_horizon_s=25.5 evicted_unread_reachable=1

  * `lost_the_race=0` over 127 copy-outs. A copy reads a staging slot the ring
    may hand out at any moment; a hand-out bumps the slot's generation, so
    re-asking `is_live` after the copy answers exactly "did anyone take this
    slot while I was reading it". Zero says the 16-slot ring covers a copy
    with room to spare, which is what sizing staging in TRANSFERS predicts.
  * `no_region=0`, `refused=0`: the bound never bit.
  * `warm=96 cold=31`. The destination pool WARMS. The retired
    `RestoreDestinationArena` measured `arena.warm=0` on every boot in the
    prior receipt, because a pool holding restores alone gets a region back
    only when a restored entry is dropped and live conversations drop none.
    Sharing one pool with retention is what fixed it.
  * `reuse_horizon_s=25.5` — MEASURED, the longest interval after which a read
    actually landed. It is the window the working set is defined over, and it
    is roughly the drive's 12 s think-time plus a turn, which is what it
    should be.
  * `demand_entries` tracked 8 → 31 → 16 as conversations arrived and ended,
    and `working_set` 7 → 29 → 14 under it. The cap follows both directions.
  * `evicted_unread_reachable=1` over 88 requests: one entry was lost that
    something could still have asked for, and the loop's error term absorbed
    it by widening the target. Nonzero is how an under-sized cap is visible.
  * `below_floor=0`: the one-chunk floor culled nothing HERE, and that is a
    fact about this workload rather than about the floor. Every savepoint this
    drive writes lands at a `chunk_prefill` boundary of a ~6,000-token prompt
    or past it, so none of them covers less than one chunk. The floor bites on
    conversations whose whole turn is under 2,048 tokens, which this harness
    does not produce; it needs its own measurement.
  * The disk tier's spill gate now names WHICH refusal fired
    (`spill_gate_reject_{slot_reused,dominated,unmeasured,not_worth_it}`
    beside the total). It reads zero throughout here because the tier is
    inert, but the prior A/B's `spill_gate_reject=25-26` against 68-77 flushes
    could not be attributed at all, and the next disk run will not have that
    problem.

LOSSLESS, DEMONSTRATED ON THE CARD
-----------------------------------
Not asserted. A 96-row savepoint at mixed fp32 / bf16 / fp16, staged from the
device into a pinned ring slot exactly as `snapshot_recurrent_row` does, then
copied out into a pageable region and left through sixteen laps of a four-slot
ring:

  sha256 device source  = 02fb4ac5cc23109ad12a81af9f21105647140f5f99010f3ff6c3007117f49173
  sha256 after pageable = 02fb4ac5cc23109ad12a81af9f21105647140f5f99010f3ff6c3007117f49173
  mismatched rows       = 0        BIT-IDENTICAL

`tests/test_savepoint_pageable_retention.py` pins the same property per row and
per dtype off-GPU.

WHAT THIS DOES NOT SAY
-----------------------
  * **Not a long-context result.** ~6,300-token prompts are the worst case for
    a retention tier, because a savepoint's cost is flat in the prompt and its
    value is linear in it. The long-context mix is the measurement of record
    and it is pending. Nothing here should be read as a final magnitude, and
    no constant in the implementation was chosen against these numbers.
  * **Two replicates of the branch, not a distribution.** D and D2 agree on
    hit rate (89.4 / 89.3) and decode (48.7 / 49.0) and disagree on TTFT
    (0.272 / 0.181). Direction is robust; magnitude is not.
  * **The disk tier is untouched and unmeasured here.** It stayed inert in
    every arm by design. Its budget is still calibrated to the drive's
    capability rather than to serving headroom (#2315), and its flush band
    still ranks by recency alone; both are named in the code where a reader
    will look for them and neither is fixed, because whether bytes reach a
    drive at all is decided one tier up and a properly sized pageable pool
    does not generate that traffic.
  * `evicted_unread_reachable=1` is not zero. The loop corrected it, but a
    workload whose reuse interval has a longer tail than 25.5 s would clip
    that tail before correcting, once per clipped conversation.
  * One drafter OOM self-heal fired during arm B (the engine narrowed the
    prefill chunk and continued); 0 request errors in every arm.
  * The RSS column was reported as unexplained and is now explained by the
    allocator's power-of-two rounding — see the section above. The probe was
    designed to fail (a pageable control of the same size, measured the same
    way) and the control came out at 1.000.

WHAT WAS BUILT AFTER THIS RUN, AND WHAT IS THEREFORE UNMEASURED
----------------------------------------------------------------
Everything above was measured on the pinned/pageable split alone. Five things
landed after it, on the owner's instruction to build the whole thing before
measuring any of it again, and NONE of them is in the table:

  * the flush band's budget closed against the spill fence's residual rather
    than the drive's capability (#2315);
  * write-back — the band and the allocator both refuse an entry whose
    conversation has ended, preferring a drop to a spill;
  * prefetch — the restore starts when the match resolves and is claimed
    without waiting;
  * retention regions backed by a memfd another process can map by id;
  * one fused ``_foreach_copy_`` out of staging instead of one per row.

The last of those was compared end to end here and the comparison FAILED to
resolve: one replicate each read 0.197 s / 50.6 tok/s fused against 0.321 s /
47.3 tok/s per-row, direction favouring the fused copy, while the per-row arm
alone spans 0.181-0.321 s across two runs of identical code. A spread wider
than the effect cannot separate a contention change from run-to-run variance,
so it is folded in on its mechanism — one GIL release instead of one per slab,
beside an engine loop that must hold the GIL to launch a step — and not on
that pair. Its bit-identity across a mixed fp32/bf16/fp16 list IS established
on the card: sha256 26a83b3f… on both sides of device → pinned → pageable → 16
ring laps, 0 mismatched rows.

Two harness facts bound any future comparison on this drive, and both were
found after the arms above ran. MTP accept rate on this filler is 0.374
against an owner baseline of ~200 tok/s where these arms read 47-51, because
the filler is one sentence repeated and the drafter predicts it badly — so the
tok/s here are comparable BETWEEN arms and are not production figures. And the
prompts are ~6,300 tokens, which is the caveat this file opens with.

VERDICT
-------
The middle tier does what it was built for. Retention leaves the page-locked
ledger, the staging ring collapses to the transfers it actually stages, and
sizing retention by the observed working set makes the split a win on total
savepoint memory as well as on the ledger that can kill the process. The disk tier is
what it should be: the overflow path for a working set that pageable RAM
cannot hold, and on this workload it is never reached.
