[tenancy before] (none)
[l2 scrub] mode='read' -- see the module docstring for the bias it carries

  shape  geometry                        thr  regs    smem blk/SM warps/SM   bank
  s27    128x128x128 1x4 st2 sw8         128   199   49152      2        8 128
  s110   128x128x128 1x4 st2 sw8         128   254   49152      2        8 128   <-- SPILLS
  s111   128x128x128 1x4 st2 sw8 gs128   128   255   50176      2        8 128x2

  in_proj_qkv k=5120 n=10240 M=2048   burst>= 50 ms, reps=5
    f111      742.86 us/call  inner=69     burst=   51.3 ms
    f27       676.62 us/call  inner=76     burst=   51.4 ms
    g111      554.40 us/call  inner=93     burst=   51.6 ms
    g27       485.27 us/call  inner=103    burst=   50.0 ms
    legA     2219.48 us/call  inner=24     burst=   53.3 ms
    legB      996.09 us/call  inner=52     burst=   51.8 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1425x  chain/s27 1.0941x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1425x [1.0870-1.2506]  chain 1.0941x [1.0175-1.2483]

  in_proj_qkv k=5120 n=10240 M=8192   burst>= 50 ms, reps=5
    f111     3099.58 us/call  inner=17     burst=   52.7 ms
    f27      2859.86 us/call  inner=18     burst=   51.5 ms
    g111     2159.24 us/call  inner=25     burst=   54.0 ms
    g27      1930.24 us/call  inner=25     burst=   48.3 ms
    legA     8805.58 us/call  inner=8      burst=   70.4 ms
    legB     3490.04 us/call  inner=15     burst=   52.4 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1102x  chain/s27 1.0803x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1102x [1.0068-1.3045]  chain 1.0803x [0.9968-1.1855]

  in_proj_z k=5120 n=6144 M=2048   burst>= 50 ms, reps=5
    f111      492.40 us/call  inner=109    burst=   53.7 ms
    f27       430.68 us/call  inner=120    burst=   51.7 ms
    g111      333.58 us/call  inner=160    burst=   53.4 ms
    g27       292.75 us/call  inner=172    burst=   50.4 ms
    legA     1504.49 us/call  inner=35     burst=   52.7 ms
    legB      614.91 us/call  inner=83     burst=   51.0 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1373x  chain/s27 1.1534x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1373x [1.0871-1.1777]  chain 1.1534x [1.0882-1.2217]

  in_proj_z k=5120 n=6144 M=8192   burst>= 50 ms, reps=5
    f111     2106.45 us/call  inner=25     burst=   52.7 ms
    f27      1962.57 us/call  inner=26     burst=   51.0 ms
    g111     1277.00 us/call  inner=41     burst=   52.4 ms
    g27      1148.09 us/call  inner=43     burst=   49.4 ms
    legA     6094.28 us/call  inner=9      burst=   54.8 ms
    legB     2085.07 us/call  inner=25     burst=   52.1 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1361x  chain/s27 1.0786x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1361x [0.9574-1.1640]  chain 1.0786x [0.9664-1.1611]

  out_proj k=6144 n=5120 M=2048   burst>= 50 ms, reps=5
    f111      504.15 us/call  inner=103    burst=   51.9 ms
    f27       481.01 us/call  inner=108    burst=   51.9 ms
    g111      338.30 us/call  inner=154    burst=   52.1 ms
    g27       294.85 us/call  inner=159    burst=   46.9 ms
    legA     1570.30 us/call  inner=32     burst=   50.2 ms
    legB      612.13 us/call  inner=85     burst=   52.0 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1358x  chain/s27 1.1156x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1358x [1.0391-1.1546]  chain 1.1156x [1.0040-1.1769]

  out_proj k=6144 n=5120 M=8192   burst>= 50 ms, reps=5
    f111     2195.17 us/call  inner=23     burst=   50.5 ms
    f27      1996.05 us/call  inner=26     burst=   51.9 ms
    g111     1284.25 us/call  inner=41     burst=   52.7 ms
    g27      1264.05 us/call  inner=45     burst=   56.9 ms
    legA     6214.14 us/call  inner=8      burst=   49.7 ms
    legB     2044.22 us/call  inner=25     burst=   51.1 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.0044x  chain/s27 1.0998x
      BEST grouped s111 vs BEST per-token s27: gemm 1.0044x [0.9369-1.1497]  chain 1.0998x [1.0086-1.1257]

  gate_proj k=5120 n=17408 M=2048   burst>= 50 ms, reps=5
    f111     1166.56 us/call  inner=45     burst=   52.5 ms
    f27      1056.48 us/call  inner=50     burst=   52.8 ms
    g111      918.71 us/call  inner=57     burst=   52.4 ms
    g27       839.47 us/call  inner=63     burst=   52.9 ms
    legA     3709.73 us/call  inner=14     burst=   51.9 ms
    legB     1712.12 us/call  inner=30     burst=   51.4 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.0803x  chain/s27 1.1194x
      BEST grouped s111 vs BEST per-token s27: gemm 1.0803x [1.0576-1.2062]  chain 1.1194x [0.9182-1.2212]

  gate_proj k=5120 n=17408 M=8192   burst>= 50 ms, reps=5
    f111     4908.13 us/call  inner=10     burst=   49.1 ms
    f27      4975.33 us/call  inner=11     burst=   54.7 ms
    g111     3675.65 us/call  inner=14     burst=   51.5 ms
    g27      3339.20 us/call  inner=15     burst=   50.1 ms
    legA    13830.02 us/call  inner=8      burst=  110.6 ms
    legB     5939.84 us/call  inner=8      burst=   47.5 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1078x  chain/s27 0.9912x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1078x [1.0596-1.2630]  chain 0.9912x [0.9413-1.1126]

  up_proj k=5120 n=17408 M=2048   burst>= 50 ms, reps=5
    f111     1165.09 us/call  inner=43     burst=   50.1 ms
    f27      1075.83 us/call  inner=49     burst=   52.7 ms
    g111      949.18 us/call  inner=57     burst=   54.1 ms
    g27       855.07 us/call  inner=62     burst=   53.0 ms
    legA     3734.53 us/call  inner=13     burst=   48.5 ms
    legB     1700.80 us/call  inner=30     burst=   51.0 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.0748x  chain/s27 1.0753x
      BEST grouped s111 vs BEST per-token s27: gemm 1.0748x [0.9647-1.1834]  chain 1.0753x [1.0558-1.2490]

  up_proj k=5120 n=17408 M=8192   burst>= 50 ms, reps=5
    f111     4821.18 us/call  inner=11     burst=   53.0 ms
    f27      4460.37 us/call  inner=12     burst=   53.5 ms
    g111     3679.01 us/call  inner=14     burst=   51.5 ms
    g27      3478.06 us/call  inner=16     burst=   55.6 ms
    legA    13903.23 us/call  inner=8      burst=  111.2 ms
    legB     5949.18 us/call  inner=8      burst=   47.6 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.0623x  chain/s27 1.0714x
      BEST grouped s111 vs BEST per-token s27: gemm 1.0623x [0.9190-1.2307]  chain 1.0714x [1.0251-1.2602]

  down_proj k=17408 n=5120 M=2048   burst>= 50 ms, reps=5
    f111     1449.72 us/call  inner=35     burst=   50.7 ms
    f27      1342.90 us/call  inner=33     burst=   44.3 ms
    g111      917.86 us/call  inner=52     burst=   47.7 ms
    g27       829.94 us/call  inner=49     burst=   40.7 ms
    legA     4022.68 us/call  inner=13     burst=   52.3 ms
    legB     1574.57 us/call  inner=33     burst=   52.0 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1059x  chain/s27 1.0651x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1059x [1.0154-1.1378]  chain 1.0651x [1.0401-1.1272]

  down_proj k=17408 n=5120 M=8192   burst>= 50 ms, reps=5
    f111     5936.81 us/call  inner=9      burst=   53.4 ms
    f27      5370.06 us/call  inner=10     burst=   53.7 ms
    g111     3621.60 us/call  inner=14     burst=   50.7 ms
    g27      3231.87 us/call  inner=16     burst=   51.7 ms
    legA    15168.13 us/call  inner=8      burst=  121.3 ms
    legB     5377.64 us/call  inner=10     burst=   53.8 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s111 gs128                                             gemm/s27 1.1448x  chain/s27 1.0944x
      BEST grouped s111 vs BEST per-token s27: gemm 1.1448x [1.1065-1.1933]  chain 1.0944x [1.0425-1.1322]

[sampler] 509 samples over 25.5 s of process life; 19.9 s of that was inside a timed block

[under load] sm clock median 210 MHz (min 210, max 210)  power median 34 W  sw_power_cap 0/396 samples  hw_slowdown 0/396
[tenancy after ] (none)   (this process is pid 982)
