HOTTEST-TIER KV REUSE ON HYBRIDS: REPLAY vs SAVEPOINT vs VRAM — 2026-09-10
===========================================================================

THE QUESTION. A hybrid's attention KV is paged and reusable; its GDN recurrent
state is not sliceable, so a cached prefix is only servable if the state at the
matched boundary can be produced. Three ways to produce it, and the goal is to
pick one on measurement rather than on taste:

  1. REPLAY (zero VRAM, zero host RAM) — recompute the state over the prefix.
  2. SAVEPOINT IN HOST RAM (zero VRAM) — store the state, DMA it back.
  3. SAVEPOINT IN VRAM (the "gold standard") — keep it on device, restore ~free.

RIG
  box 206 GPU 1, container sphyb, port 8127
  arbi-serve main 32a41cfd2 bind-mounted, image :test-latest
  /models/Qwen3.5-0.8B — GDN hybrid, 24 layers (6 paged / 18 recurrent),
    linear_num_key_heads 16, linear_key_head_dim 128, max_position 262144
  ARBI_SAVEPOINT_ENABLED=1 ARBI_PREFIX_TIER=1, --kv-cache-dtype=auto
  Geometry off this boot, not assumed:
    KV page       3,145,728 B / 256 tok;  pool 6,872 pages = 1,759,232 tok
    savepoint row    19,759,104 B  = 6.28 KV pages = 1,608 tokens of KV
    ring 16 slots (DMA STAGING), store retains 64 savepoints = 1.18 GiB host

FINDING 1 — REPLAY CANNOT BE CHEAP, AND THE REASON IS STRUCTURAL
  The GDN block consumes the RESIDUAL STREAM:

      h = self.input_layernorm(hidden, residual_buf)
      h = self.linear_attn(h, per_layer_view, batch_meta)

  so its q/k/v/beta/g at token t depend on every prior layer's output at t,
  attention layers included. The state at token N therefore requires a full
  forward over tokens 1..N. Replay is not "the recurrent layers only" — it is
  a prefill whose only savings are the K/V projections and the KV writes.
  Replay cost IS prefill cost for the span it covers.

  This is why the existing replay machinery does not generalise: it is all
  MTP verify (``gdn_verify_replay_save``, ``gdn_verify_chained``,
  ARBI_GDN_REPLAY_TRUNCATE_SCAN), spanning K<=7 tokens from an h0 the slab
  still holds. The kernel is right; the entry condition and the span are not.

FINDING 2 — THE MEASURED COMPARISON
  Prefill rate this boot, 3 cold reps per length, fresh prefix each rep so
  every one is a true miss:

      tokens    median s    us/token
        1,700      0.034        20.2
        6,784      0.114        16.9
       27,018      0.416        15.4
      101,353      2.741        27.0

  Savepoint host<->device leg, measured by the boot itself
  (``measure_savepoint_ring_transfer_us``): write 1510 us, RESTORE 1484 us.

  What a cache hit costs, by arm:

    prefix    savepoint(host)   replay 2048-tok gap   full re-prefill
    -------   ---------------   -------------------   ---------------
      1,700          1.48 ms            41.41 ms            34.38 ms
      6,784          1.48 ms            34.51 ms           114.32 ms
     27,018          1.48 ms            31.52 ms           415.83 ms
    101,353          1.48 ms            55.38 ms          2740.91 ms

VERDICT — SAVEPOINT IN HOST RAM, WHICH IS WHAT SHIPS
  * vs REPLAY: 21-37x cheaper. Replay over even a MINIMAL 2048-token gap
    costs 31-55 ms against 1.48 ms. Replay only ever competes if a savepoint
    is missing entirely, and then it is competing with re-prefill, not with a
    restore.
  * vs VRAM-RESIDENT: the gold standard saves 1.48 ms per hit and costs
    18.84 MiB of VRAM per retained row. At the shipped retention of 64 rows
    that is 1.18 GiB = 402 KV pages = 102,912 tokens = 5.8% of this pool.
    At a 101k prefix the hit is worth 2,740.91 ms, so the 1.48 ms the VRAM
    buys back is 0.05% of the win. Paying 5.8% of KV capacity for 0.05% of
    the benefit is a bad trade, and it gets worse as context grows because
    the savepoint restore is FLAT in prefix length while re-prefill is linear.

  The restore's flatness is the property that decides this. A savepoint row
  is the model's recurrent geometry — it does not grow with the prefix it
  stands for — so one 1.48 ms transfer serves a 1.7k prefix and a 101k prefix
  alike, while the alternative it replaces grows to 2.74 s.

WHAT THIS DOES NOT ESTABLISH
  * Arm 3 is COMPUTED, not built: its restore is ~0 by construction and its
    VRAM cost is exact arithmetic from the measured row and page sizes, but
    no VRAM-resident implementation was run. If someone builds one, the
    number to beat is 1.48 ms saved per hit.
  * One model, one card, TP1. The 27B's row is 154,927,104 B (7.8x this
    one's) and its KV page 4,734,976 B, so the ratios move; the ARGUMENT does
    not, because both terms scale with the same recurrent geometry.
  * Whether the 1.48 ms sits on the critical path or is overlapped with other
    admission work was not measured. If overlapped, the host arm wins by more.
  * Nothing here measures the pair rule's OTHER failure mode — a savepoint
    that exists but covers less than the radix matched.
