MORE KV RETENTION MAKES A HYBRID WORSE AT LONG CONTEXT — 2026-09-10
====================================================================

THE FINDING IN ONE LINE. The warm KV tier works — it faults thousands of
prefix pages back out of pinned host RAM — and on a hybrid under scarcity that
makes the 64k+ band WORSE, because savepoint_admission discards every restored
match whose paired recurrent savepoint is gone. The KV side retains 907 tier
slots; the savepoint side retains 16 ring slots. Nobody sized that asymmetry on
purpose.

RIG
  box 206 GPU 1, container sp128, port 8124
  arbi-serve main f26956b2c, bind-mounted; image :test-latest
  Qwen3.8-27B-exl3-4.0bpw, --kv-cache-dtype=tkv TKV_BITS=4, vq2 bundle
  --max-batch=8 --chunk-prefill=2048 --max-batched-tokens=2304, GMU 1.0
  971 servable pages x 256 = 248,576 tokens = 1.9x a 128k sequence
  savepoint ring 16 slots x 154,927,104 B; prefix tier arena 907 slots
  harness tools/savepoint/retention_scenarios.py --scenarios linear
    --mix 2k=6,6k=6,32k=4,128k=2 --turns 4
    --max-tokens 300 --think-time-s 12 --replay-reasoning
  Two arms, same commit, same card, fresh container and fresh nonce each:
    TIER128    ARBI_PREFIX_TIER=1
    NOTIER128  ARBI_PREFIX_TIER=0

RESULT — 64k+ BAND
                        tier ON    tier OFF
  resumed anything      1/6        0/6
  cached% median        0.0        0.0
  cached% mean          16.6       0.0
  ttft_saved%           42.3       39.6

  tier_restore_total            3160       (series absent — tier off)
  tier_arena handouts           4776       --
  tier_arena exhausted          1621       --
  hybrid_no_savepoint_fallback  1396       787
  recurrent_cap_truncated_total 1478       1384
  prefix_cache_hit_total        2874       2171

READ IT IN THIS ORDER
1. The tier restored 3160 pages. The mechanism is not broken and this is the
   first run in which that could be SEEN — tier_restore_total had no
   forwarding path until arbi-serve#2333, so on every earlier boot its series
   did not exist and its absence was not evidence of anything.
2. hybrid_no_savepoint_fallback fired 1396 times. That counter is a radix
   match with NO savepoint to pair with, which savepoint_admission answers by
   discarding the KV match too and re-prefilling from token 0 — the most
   expensive single event in the system.
3. The tier arm has MORE total losses than the off arm: 1396 vs 787. Restoring
   KV that has no surviving savepoint partner MANUFACTURES unpaired matches.
   The tier is doing work the pair rule then throws away, and it creates the
   very condition that throws it away.
4. So on a hybrid under scarcity, better KV retention alone is not neutral. It
   is negative on the counter that matters, and it buys 1/6 instead of 0/6.

WHY: AN ASYMMETRY NOBODY SIZED
  KV side, warm tier:   907 slots x 4,734,976 B  = 4.29 GiB pinned
  savepoint side, ring:  16 slots x 154,927,104 B = 2.31 GiB pinned
  A savepoint row is 33x a KV page, so equal pinned bytes buy 57x fewer
  savepoint rows. The recurrent half was never given a tier of its own, so the
  two halves of a pair retain at wildly different depths and the pair rule
  resolves every disagreement by discarding BOTH.

THE CONTROL THAT ISOLATES IT
  Qwen3.5-0.8B GDN hybrid, 6868 pages = 1,758,208 tokens = 13.4x a 128k
  sequence. Nothing is ever evicted, so the pair sink never fires, and the same
  128k band caches 99.3% median with TTFT 6.19 s -> 0.19 s
  (docs/receipts/long-context-128k-2026-09-09.txt). 128k is not inherently
  broken. It breaks exactly when eviction starts.

WHAT THIS SAYS ABOUT THE LEVER
  Raising KV capacity or improving the KV victim policy cannot lift long
  context on a hybrid on its own — whatever KV is retained is discarded unless
  its savepoint survives too. The candidate levers are, in order of how
  directly they attack the measured cause:
    a. retain the PAIR as a unit — a savepoint and its KV pages are kept or
       dropped together, so a restore can never be orphaned;
    b. give the savepoint side its own tier, so its retention depth is
       comparable to the KV side's;
    c. let the pair rule serve a PARTIAL resume rather than discarding both
       halves.
  None of these is measured here. They are named because the measurement
  points at them, not because they are known to work.

WHAT THIS DOES NOT ESTABLISH
  * One card, one model, TP1, ONE drive per arm. No interval, no MDE — these
    are single observations. The 1/6 vs 0/6 difference is one turn and must
    not be quoted as an effect size.
  * Nothing about DENSE models, which have no recurrent half and therefore no
    pair and no sink. The tier's value there is untouched by this result.
  * Nothing about the cold/disk tier, which was off in both arms.
  * The earlier long-context receipt put this band at 0/6 with the tier off.
    That agrees with the off arm here, but it was a different commit and is
    not treated as a third data point.

A METHOD NOTE WORTH KEEPING
  A first attempt at this A/B omitted --max-tokens 300 --think-time-s 12
  --replay-reasoning, the parameters the run of record used. The harness then
  generated uncapped: single replies of 82,812 and 116,870 tokens, which wrapped
  the 16-slot ring on their own and made each arm exceed an hour. That is a
  different workload, not a noisier version of the same one, and its numbers
  were discarded rather than reported. Reproduce the reference parameters or do
  not compare to the reference number.
