GPU kernel lane: the RSS ceiling red on the EXL3 swap test -- box 206, 2026-09-08
================================================================================

THE RED
-------
`test-gpu` (GPU kernel tests, in-image) has been the only thing between main
and `publish` since the e2e lane came back. Its final line names one sub-lane:

    losslessness-gate rc=0  kernels rc=1  sync-ratchet rc=0
    host-gate-audit rc=0 (DEFERRED)  lora-e2e rc=0 (DEFERRED)  multi-gpu rc=0 (DEFERRED)

and `kernels` carries exactly one failure (run 34254074651, job 102219105299):

    RSS CEILING EXCEEDED by tests/test_exl3_weights.py::
      test_exl3_swap_replaces_every_quantized_linear:
      peak 4258 MiB > ceiling 4096 MiB on worker main
      (this test: rss 3238 -> 3238 MiB, growth +0 MiB; peak 4258 MiB)
    1 failed, 236 passed, 3 skipped, 19742 deselected in 70.63s

`growth +0` is the first thing worth reading: the test RETAINS nothing at the
end of the call. What crossed is `peak` -- the working set it holds while it
runs, on top of a worker baseline of 3238 MiB it did not create.

WHERE THE WORKING SET GOES
--------------------------
Measured on box 206, in registry.arbi.work/arbi-serve:test-4a2f15c, by walking
`Qwen3Model.from_safetensors`'s own phases and reading /proc/self/status VmHWM
between them. `device="cuda:0"`, the checkpoint the test loads
(/mnt/k8scache/models/Qwen3-0.6B-exl3, model.safetensors = 724.7 MiB):

    phase                                    rss MiB    VmHWM MiB
    start                                        662          662
    after init_distributed_environment            662          662
    after SafetensorsCollection(model_dir)        662          662
    after prepare_dims_for_quant                  662          662
    after build() under torch.device("meta")      664          664
    after _apply_quant_and_materialize           1258         1258
    after load_model_weights                     1556         1850

Two phases spend it, in roughly equal halves:

  * `_apply_quant_and_materialize` -- the QUANT SWAP, +594 MiB. This is the
    EXL3 backend reading the trellis to bind 197 linears. It is the subject of
    the test.
  * `load_model_weights` -- the WEIGHT FILL, a further +592 MiB of peak. It
    stages the shard into host memory to copy it to the card.

The assertions were already satisfied at the FIRST of those. Printed by the
probe immediately after the swap, before the fill ran:

    after swap ONLY: n_exl3=197 n_dense=0
                     lm_head=EXL3ReplicatedLinear
                     q_proj=EXL3ColumnParallelLinear
                     o_proj=EXL3RowParallelLinear

which is, name for name, everything the test asserts. The fill is half the
working set and none of the evidence: this test reads no weight VALUE.

THE FIX, AND WHY IT IS NOT A TUNING KNOB
----------------------------------------
Build on `device="meta"`. The loader already meta-constructs and then
materializes to the device it is handed, so `meta` runs config parse,
topology validation, quant prep and the full quant swap unchanged, and the
fill has nothing real to fill. Same public entry point, no spy, no new flag,
no monkeypatch, and no change to `DEFAULT_CEILING_MIB`.

    build                          peak delta   growth   n_exl3 / n_dense
    device="cuda:0"                 ~1143 MiB   +125.3 MiB   197 / 0
    device="meta"                    ~608 MiB     +3.1 MiB   197 / 0

Both numbers move, and the second one matters beyond this test: on a lane that
runs in ONE process, 125 MiB this test used to retain was 125 MiB of baseline
charged to every test that followed it.

NULL CONTROL -- THE CHEAPER BUILD IS NOT A FREE GREEN
-----------------------------------------------------
A build that stopped swapping would still have to fail. Same meta path, same
loader, two checkpoints:

    Qwen3-0.6B-exl3   meta build -> n_exl3= 197  n_dense=   0   (asserts 197/0: PASS)
    Qwen3-0.6B        meta build -> n_exl3=   0  n_dense= 197   (asserts 197/0: FAIL)

The dense checkpoint drives every counter to the opposite end. The assertions
keep their full discriminating power on `meta`.

WHY NOT THE OTHER TWO REMEDIES THE CEILING OFFERS
-------------------------------------------------
  * `@pytest.mark.rss_ceiling_mib(N)` is right for a test whose working set IS
    the checkpoint it exists to load and verify -- which is why
    tests/test_dflash_exl3_forward_gpu.py carries one. This test verifies a
    module GRAPH, so spending a checkpoint-sized allowance on it would buy
    coverage it does not have.
  * Raising DEFAULT_CEILING_MIB answers a lane whose floor moved. The floor
    here did not move; one test's working set was twice what its assertions
    needed.

Real EXL3 weights on a real device, decoded and checked for coherence, remain
covered by tests/test_exl3_live_qwen3_generation.

THE LANE PRINTED NO RANKING
---------------------------
The ceiling's failure text ends "read the --rss-top ranking". scripts/ci/
cpu_suite.sh passes `--rss-top=10`; scripts/ci/gpu_lane.sh's `run_kernels`
passed nothing, so the lane that ENFORCES the ceiling was the one lane whose
red arrived with no way to tell a test that grew from a lane whose floor rose
under it. `--rss-top=25` is added there.

THE SECOND RED, WHICH `-x` HAD HIDDEN
--------------------------------------
The kernel lane runs `pytest tests/ -x`. `-x` stops at the FIRST failure, and
the ceiling failure above is at 23% of the selection, so for as long as it
stood nothing after it ran in CI at all. Running the same selector on 206
WITHOUT `-x` (and with the ceiling disabled, so the known red does not stop
the walk) is what shows the rest of the lane:

    930 selected, 912 ran, 19 skipped -- 30m39s
    1 failed: tests/test_named_pools.py::test_snapshot_emits_one_shot_warning_at_80pct
              AssertionError: expected 1 WARNING, got 0: []

CI's own runs report `239 selected` against this run's `930`, which is the
same fact from the other side: `-x` was ending the lane a quarter of the way
in. 240/1030 is the 23%.

That second failure is NOT an ordering artefact. It reproduces alone:

    python -m pytest tests/test_named_pools.py::\
      test_snapshot_emits_one_shot_warning_at_80pct -q -m gpu
    -> 1 failed in 0.83s

`NamedMemPool.snapshot()` reads its two numbers from
`_allocated_and_reserved_for_pool`, which takes both sums from one segment
walk. The test patched `_bytes_for_pool` and `_reserved_for_pool`. Both of
those still exist and still serve the two properties, so `monkeypatch.setattr`
found its targets and raised nothing -- and the snapshot path does not call
either. The pool reported its real zero bytes, 80% of the target was never
crossed, and the one-shot guard was never exercised.

Driving real bytes instead removes the failure mode rather than re-pointing
it. A 9 MiB tensor allocated inside the pool's own `use()` window and held
live across both snapshots:

    snapshot1: caching_allocator_allocated_bytes=9437184
               caching_allocator_reserved_bytes=20971520
               size_target_bytes=10485760  peak_allocated_bytes=9437184
    WARN count over TWO snapshots: 1
      "NamedMemPool('warn_pool'): allocated=9437184 crossed 80% of
       size_target_bytes=10485760 ..."

    tests/test_named_pools.py -m gpu -> 13 passed in 12.00s

WHAT THE LANE'S RSS RANKING SAYS
--------------------------------
With `--rss-top=25` on (the flag this change adds), the full-lane ranking is
led by tests that carry their own measured allowance, not by a drifting floor:

     +1280.1 MiB  rss end 5120 MiB  test_in_place_register.py::
                                    test_register_param_in_place_peak_vram_below_1_2x_baseline
      +512.3 MiB  rss end 2179 MiB  runtime/test_preload_overlap.py::
                                    test_preload_overlap_reduces_wake_latency
      +136.3 MiB  peak    4268 MiB  test_dflash_exl3_forward_gpu.py::
                                    test_exl3_dflash_loader_binds_checkpoint_faithfully[shipped]

The EXL3 swap test does not appear in the top 25 after the change; before it,
it sat at +125.3 MiB of retention with a peak over the ceiling.

THE THIRD RED: A CEILING SITTING INSIDE THE LANE'S OPERATING BAND
------------------------------------------------------------------
Re-running the kernel selector WITH the ceiling enforced and the EXL3 fix in
place clears the original red and surfaces a new one:

    2 failed, 910 passed, 19 skipped -- 7m19s
      test_named_pools.py::test_snapshot_emits_one_shot_warning_at_80pct  (above)
      test_verify_sharded_gpu.py::test_recovery_bonus_byte_identical_full_vocab[32768]
        rss 4110 MiB > ceiling 4096 MiB
        (this test: rss 4074 -> 4110 MiB, growth +36 MiB; peak 4110 MiB)

The EXL3 swap test no longer appears anywhere in the top 25.

That second failure is NOT this test's doing, and the two runs prove it by
putting the SAME test on both sides of the rule. A test is charged only when
it ENDS above the ceiling having STARTED below it:

    run                          in_place_register   verify_sharded        charged?
                                 (rss end)           (rss start -> end)
    before the EXL3 fix          5120 MiB            4913 -> 5021 MiB      NO  (started above)
    after  the EXL3 fix          4752 MiB            4074 -> 4110 MiB      YES (straddles)

Nothing about `verify_sharded` changed. What changed is the LANE: freeing
memory upstream lowered every resting point after it, and a test that used to
begin life above the line now begins below it and crosses. Its own growth is
+36 MiB. It is named for the lane's floor, not for its own working set.

That is the failure mode of a ceiling placed INSIDE the band a lane normally
occupies rather than above it. The band, from the same ranking:

    4752 MiB  test_in_place_register.py (carries its own mark: 5120)
    4806 MiB  test_mamba2_chunked_prefill_kernels_gpu.py   (no mark)
    4809 MiB  test_mamba2_fold_emit_gpu.py                 (no mark, lane's highest)
    4180 MiB  test_dflash_exl3_forward_gpu.py[shipped]     (carries its own mark: 5120)

Three of those sit above 4096 and go uncharged only because they start above
it. A ceiling that a lane crosses and re-crosses in its ordinary course does
not measure "this test materialised too much"; it measures which test was
holding the parcel when the music stopped, and it makes a MEMORY SAVING look
like a regression.

THE VERDICT DOES NOT BELONG ON THIS LANE AT ALL
------------------------------------------------
Raising the number was the wrong read, and the derivation is what says so.
The threshold is not a property of a test; it is a PER-XDIST-WORKER budget:
scripts/ci/xdist_workers.sh reserves CPU_SUITE_WORKER_MIB (2048) per worker,
and tests/test_cpu_test_rss_ceiling.py asserts the ceiling sits above that
reservation by "the working set one test may add on top". 4096 is 2048 plus
headroom, for one of four workers sharing a box.

The kernel lane has no workers:

    grep -cE '-n [0-9]+|-n auto|xdist' scripts/ci/gpu_lane.sh   ->  0
    grep -cE '-n [0-9]+|-n auto|xdist' scripts/ci/cpu_suite.sh  ->  2

It is one process with the whole 93 GB box. It inherits the plugin only
because tests/conftest.py sets `pytest_plugins = ("pytester",
"tests._rss_ceiling")`, so every pytest anywhere gets it. Applying a
per-worker slice to a single process that owns the box is a category error,
and 4096 MiB is ~4% of that box: the number cannot tell "this test endangers
the machine" from "the lane is where it always is".

The corroboration is in the tree. ALL THREE `rss_ceiling_mib` marks are on
tests of this lane -- test_in_place_register.py, test_dflash_exl3_forward_gpu.py
and test_host_sync_ratchet_live.py. Three earlier rounds of the same
whack-a-mole, and `verify_sharded` would have been the fourth.

So the lane exports ARBI_CPU_TEST_RSS_CEILING_MIB=0, which drops the VERDICT
and keeps the measurement and the --rss-top ranking, and DEFAULT_CEILING_MIB
stays at 4096 untouched -- which also takes the CPU suite out of this change
entirely and makes the two 19,123-pass runs above no longer load-bearing.

THE LANE, GREEN, END TO END
---------------------------
`bash scripts/ci/gpu_lane.sh` -- the real lane script, not a selector -- on
box 206, both cards idle, all three fixes in:

    losslessness-gate rc=0  kernels rc=0  sync-ratchet rc=0
    host-gate-audit rc=0 (DEFERRED)  lora-e2e rc=0 (DEFERRED)
    multi-gpu rc=0 (DEFERRED)

    sync-ratchet: 2 test(s) executed, all passed (rc=0)
      [cudagraphs] 395.99s   [eager] 29.7s

Against the failing CI run this replaces, where the same line read
`kernels rc=1`.
