[tenancy before] (none)
[l2 scrub] mode='read' -- see the module docstring for the bias it carries

  shape  geometry                        thr  regs    smem blk/SM warps/SM   bank
  s27    128x128x128 1x4 st2 sw8         128   199   49152      2        8 128
  s85    64x128x128 1x8 st2 sw8 gs128    256   128   32768      2       16 32x2
  s86    64x128x128 1x8 st2 sw8          256    78   32768      3       24 32
  s89    64x128x128 1x8 st2 sw8 gs256    256   126   32768      2       16 32x2
  s98    16x128x128 1x8 st2 sw8 gs128    256    63   20480      4       32 8x2
  s99    16x128x128 1x8 st2 sw8          256    63   20480      4       32 8
  s102   16x128x128 1x8 st2 sw8 gs256    256    63   20480      4       32 8x2

  out_proj k=6144 n=5120 M=48   burst>= 50 ms, reps=7
    f102       53.61 us/call  inner=798    burst=   42.8 ms
    f27        95.50 us/call  inner=472    burst=   45.1 ms
    f85        75.11 us/call  inner=575    burst=   43.2 ms
    f86        67.87 us/call  inner=660    burst=   44.8 ms
    f89        70.13 us/call  inner=617    burst=   43.3 ms
    f98        56.79 us/call  inner=758    burst=   43.0 ms
    f99        54.21 us/call  inner=815    burst=   44.2 ms
    g102       31.60 us/call  inner=1427   burst=   45.1 ms
    g27        68.04 us/call  inner=665    burst=   45.2 ms
    g85        49.50 us/call  inner=901    burst=   44.6 ms
    g86        41.71 us/call  inner=1098   burst=   45.8 ms
    g89        44.47 us/call  inner=1023   burst=   45.5 ms
    g98        34.53 us/call  inner=1298   burst=   44.8 ms
    g99        28.17 us/call  inner=1617   burst=   45.6 ms
    legA       44.37 us/call  inner=1029   burst=   45.7 ms
    legB      117.45 us/call  inner=405    burst=   47.6 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.6125x  chain/s27 0.7116x
      s85  gs128      bank vs s86  1.1866x [1.1642-1.1945]  gemm/s27 0.7265x  chain/s27 0.7872x
      s89  gs256      bank vs s86  1.0682x [1.0447-1.0708]  gemm/s27 0.6540x  chain/s27 0.7333x
      s99  per-token                                         gemm/s27 0.4127x  chain/s27 0.5672x
      s98  gs128      bank vs s99  1.2283x [1.2223-1.3256]  gemm/s27 0.5069x  chain/s27 0.5954x
      s102 gs256      bank vs s99  1.1243x [1.1201-1.1327]  gemm/s27 0.4642x  chain/s27 0.5612x
      BEST grouped s102 vs BEST per-token s99: gemm 1.1243x [1.1201-1.1327]  chain 0.9888x [0.9626-1.0032]

  out_proj k=6144 n=5120 M=64   burst>= 50 ms, reps=7
    f102       73.00 us/call  inner=629    burst=   45.9 ms
    f27        96.46 us/call  inner=502    burst=   48.4 ms
    f85        73.82 us/call  inner=633    burst=   46.7 ms
    f86        69.21 us/call  inner=691    burst=   47.8 ms
    f89        67.95 us/call  inner=681    burst=   46.3 ms
    f98        74.83 us/call  inner=627    burst=   46.9 ms
    f99        72.87 us/call  inner=658    burst=   47.9 ms
    g102       49.79 us/call  inner=992    burst=   49.4 ms
    g27        68.63 us/call  inner=721    burst=   49.5 ms
    g85        50.46 us/call  inner=936    burst=   47.2 ms
    g86        41.92 us/call  inner=1170   burst=   49.0 ms
    g89        44.84 us/call  inner=1089   burst=   48.8 ms
    g98        51.39 us/call  inner=961    burst=   49.4 ms
    g99        45.74 us/call  inner=1077   burst=   49.3 ms
    legA       51.41 us/call  inner=953    burst=   49.0 ms
    legB      120.69 us/call  inner=422    burst=   50.9 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.6116x  chain/s27 0.7180x
      s85  gs128      bank vs s86  1.2013x [1.1908-1.2096]  gemm/s27 0.7353x  chain/s27 0.7659x
      s89  gs256      bank vs s86  1.0697x [1.0573-1.1148]  gemm/s27 0.6532x  chain/s27 0.7039x
      s99  per-token                                         gemm/s27 0.6668x  chain/s27 0.7557x
      s98  gs128      bank vs s99  1.1242x [1.1076-1.1248]  gemm/s27 0.7493x  chain/s27 0.7746x
      s102 gs256      bank vs s99  1.0885x [1.0759-1.1270]  gemm/s27 0.7265x  chain/s27 0.7564x
      BEST grouped s89 vs BEST per-token s86: gemm 1.0697x [1.0573-1.1148]  chain 0.9817x [0.9332-0.9851]

  out_proj k=6144 n=5120 M=2048   burst>= 50 ms, reps=7
    f102     1166.03 us/call  inner=44     burst=   51.3 ms
    f27       458.89 us/call  inner=114    burst=   52.3 ms
    f85       609.79 us/call  inner=86     burst=   52.4 ms
    f86       511.95 us/call  inner=100    burst=   51.2 ms
    f89       571.80 us/call  inner=90     burst=   51.5 ms
    f98      1194.18 us/call  inner=42     burst=   50.2 ms
    f99      1064.11 us/call  inner=48     burst=   51.1 ms
    g102     1001.87 us/call  inner=51     burst=   51.1 ms
    g27       308.17 us/call  inner=171    burst=   52.7 ms
    g85       433.81 us/call  inner=120    burst=   52.1 ms
    g86       364.58 us/call  inner=140    burst=   51.0 ms
    g89       419.87 us/call  inner=127    burst=   53.3 ms
    g98      1029.98 us/call  inner=51     burst=   52.5 ms
    g99       930.11 us/call  inner=57     burst=   53.0 ms
    legA     1693.82 us/call  inner=34     burst=   57.6 ms
    legB      643.11 us/call  inner=81     burst=   52.1 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 1.1708x  chain/s27 1.1038x
      s85  gs128      bank vs s86  1.1820x [1.1353-1.2938]  gemm/s27 1.3917x  chain/s27 1.3276x
      s89  gs256      bank vs s86  1.1521x [1.1185-1.2343]  gemm/s27 1.3348x  chain/s27 1.2317x
      s99  per-token                                         gemm/s27 3.0205x  chain/s27 2.3240x
      s98  gs128      bank vs s99  1.0943x [1.0365-1.1988]  gemm/s27 3.3154x  chain/s27 2.5574x
      s102 gs256      bank vs s99  1.1138x [1.0084-1.1584]  gemm/s27 3.3642x  chain/s27 2.5279x
      BEST grouped s89 vs BEST per-token s27: gemm 1.3348x [1.3095-1.4946]  chain 1.2317x [1.0573-1.3461]

  gate_proj k=5120 n=17408 M=48   burst>= 50 ms, reps=7
    f102      107.36 us/call  inner=434    burst=   46.6 ms
    f27       123.74 us/call  inner=399    burst=   49.4 ms
    f85        94.97 us/call  inner=424    burst=   40.3 ms
    f86        86.26 us/call  inner=566    burst=   48.8 ms
    f89        96.47 us/call  inner=508    burst=   49.0 ms
    f98       110.20 us/call  inner=428    burst=   47.2 ms
    f99       103.36 us/call  inner=467    burst=   48.3 ms
    g102       86.66 us/call  inner=570    burst=   49.4 ms
    g27        95.45 us/call  inner=525    burst=   50.1 ms
    g85        66.70 us/call  inner=728    burst=   48.6 ms
    g86        58.37 us/call  inner=857    burst=   50.0 ms
    g89        63.24 us/call  inner=790    burst=   50.0 ms
    g98        89.85 us/call  inner=553    burst=   49.7 ms
    g99        81.73 us/call  inner=614    burst=   50.2 ms
    legA      104.38 us/call  inner=483    burst=   50.4 ms
    legB      483.02 us/call  inner=104    burst=   50.2 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.6147x  chain/s27 0.6954x
      s85  gs128      bank vs s86  1.1395x [1.0726-1.1628]  gemm/s27 0.6988x  chain/s27 0.7653x
      s89  gs256      bank vs s86  1.0835x [1.0010-1.1069]  gemm/s27 0.6641x  chain/s27 0.7798x
      s99  per-token                                         gemm/s27 0.8563x  chain/s27 0.8404x
      s98  gs128      bank vs s99  1.0986x [1.0909-1.1198]  gemm/s27 0.9407x  chain/s27 0.8939x
      s102 gs256      bank vs s99  1.0649x [1.0516-1.0666]  gemm/s27 0.9052x  chain/s27 0.8732x
      BEST grouped s89 vs BEST per-token s86: gemm 1.0835x [1.0010-1.1069]  chain 1.1234x [1.0363-1.1881]

  gate_proj k=5120 n=17408 M=64   burst>= 50 ms, reps=7
    f102      130.56 us/call  inner=378    burst=   49.4 ms
    f27       126.06 us/call  inner=392    burst=   49.4 ms
    f85        94.57 us/call  inner=505    burst=   47.8 ms
    f86        88.08 us/call  inner=555    burst=   48.9 ms
    f89        91.04 us/call  inner=527    burst=   48.0 ms
    f98       133.96 us/call  inner=364    burst=   48.8 ms
    f99       125.18 us/call  inner=398    burst=   49.8 ms
    g102      108.84 us/call  inner=461    burst=   50.2 ms
    g27        99.91 us/call  inner=521    burst=   52.1 ms
    g85        69.25 us/call  inner=712    burst=   49.3 ms
    g86        59.28 us/call  inner=844    burst=   50.0 ms
    g89        64.60 us/call  inner=780    burst=   50.4 ms
    g98       113.07 us/call  inner=446    burst=   50.4 ms
    g99       101.77 us/call  inner=494    burst=   50.3 ms
    legA      175.34 us/call  inner=293    burst=   51.4 ms
    legB      484.59 us/call  inner=104    burst=   50.4 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.5972x  chain/s27 0.6961x
      s85  gs128      bank vs s86  1.1684x [1.0410-1.2515]  gemm/s27 0.6980x  chain/s27 0.7554x
      s89  gs256      bank vs s86  1.0739x [0.9872-1.1155]  gemm/s27 0.6498x  chain/s27 0.7215x
      s99  per-token                                         gemm/s27 1.0147x  chain/s27 0.9924x
      s98  gs128      bank vs s99  1.1118x [1.1005-1.1271]  gemm/s27 1.1346x  chain/s27 1.0577x
      s102 gs256      bank vs s99  1.0704x [1.0554-1.0978]  gemm/s27 1.0974x  chain/s27 1.0133x
      BEST grouped s89 vs BEST per-token s86: gemm 1.0739x [0.9872-1.1155]  chain 1.0274x [1.0068-1.0800]

  gate_proj k=5120 n=17408 M=2048   burst>= 50 ms, reps=7
    f102     3105.02 us/call  inner=17     burst=   52.8 ms
    f27      1124.60 us/call  inner=46     burst=   51.7 ms
    f85      1533.69 us/call  inner=35     burst=   53.7 ms
    f86      1293.87 us/call  inner=40     burst=   51.8 ms
    f89      1436.72 us/call  inner=36     burst=   51.7 ms
    f98      3294.77 us/call  inner=16     burst=   52.7 ms
    f99      3027.67 us/call  inner=17     burst=   51.5 ms
    g102     2905.37 us/call  inner=18     burst=   52.3 ms
    g27       911.03 us/call  inner=61     burst=   55.6 ms
    g85      1286.29 us/call  inner=43     burst=   55.3 ms
    g86      1035.30 us/call  inner=50     burst=   51.8 ms
    g89      1192.46 us/call  inner=43     burst=   51.3 ms
    g98      3021.64 us/call  inner=17     burst=   51.4 ms
    g99      2682.13 us/call  inner=19     burst=   51.0 ms
    legA     3846.82 us/call  inner=15     burst=   57.7 ms
    legB     1772.43 us/call  inner=29     burst=   51.4 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 1.1507x  chain/s27 1.1397x
      s85  gs128      bank vs s86  1.2316x [1.1402-1.2860]  gemm/s27 1.3774x  chain/s27 1.3638x
      s89  gs256      bank vs s86  1.1550x [1.0894-1.1957]  gemm/s27 1.3399x  chain/s27 1.2705x
      s99  per-token                                         gemm/s27 2.9520x  chain/s27 2.7169x
      s98  gs128      bank vs s99  1.1357x [1.0835-1.1593]  gemm/s27 3.2718x  chain/s27 2.9460x
      s102 gs256      bank vs s99  1.0966x [1.0427-1.1033]  gemm/s27 3.1704x  chain/s27 2.7503x
      BEST grouped s89 vs BEST per-token s27: gemm 1.3399x [1.1703-1.4206]  chain 1.2705x [1.2261-1.3569]

  down_proj k=17408 n=5120 M=48   burst>= 50 ms, reps=7
    f102      113.88 us/call  inner=417    burst=   47.5 ms
    f27       215.29 us/call  inner=230    burst=   49.5 ms
    f85       169.22 us/call  inner=290    burst=   49.1 ms
    f86       140.32 us/call  inner=352    burst=   49.4 ms
    f89       154.03 us/call  inner=317    burst=   48.8 ms
    f98       122.21 us/call  inner=399    burst=   48.8 ms
    f99       102.46 us/call  inner=477    burst=   48.9 ms
    g102       89.24 us/call  inner=562    burst=   50.2 ms
    g27       189.13 us/call  inner=266    burst=   50.3 ms
    g85       137.13 us/call  inner=355    burst=   48.7 ms
    g86       114.60 us/call  inner=440    burst=   50.4 ms
    g89       123.56 us/call  inner=402    burst=   49.7 ms
    g98        95.59 us/call  inner=510    burst=   48.8 ms
    g99        80.83 us/call  inner=623    burst=   50.4 ms
    legA      107.40 us/call  inner=464    burst=   49.8 ms
    legB      488.34 us/call  inner=103    burst=   50.3 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.6026x  chain/s27 0.6512x
      s85  gs128      bank vs s86  1.2068x [1.1665-1.2464]  gemm/s27 0.7278x  chain/s27 0.7883x
      s89  gs256      bank vs s86  1.0876x [1.0493-1.1404]  gemm/s27 0.6554x  chain/s27 0.7172x
      s99  per-token                                         gemm/s27 0.4252x  chain/s27 0.4759x
      s98  gs128      bank vs s99  1.1943x [1.1826-1.2473]  gemm/s27 0.5058x  chain/s27 0.5660x
      s102 gs256      bank vs s99  1.1037x [1.0959-1.1200]  gemm/s27 0.4713x  chain/s27 0.5284x
      BEST grouped s102 vs BEST per-token s99: gemm 1.1037x [1.0959-1.1200]  chain 1.1149x [1.0876-1.1555]

  down_proj k=17408 n=5120 M=64   burst>= 50 ms, reps=7
    f102      167.01 us/call  inner=294    burst=   49.1 ms
    f27       217.56 us/call  inner=229    burst=   49.8 ms
    f85       168.24 us/call  inner=292    burst=   49.1 ms
    f86       141.15 us/call  inner=351    burst=   49.5 ms
    f89       152.42 us/call  inner=321    burst=   48.9 ms
    f98       171.90 us/call  inner=285    burst=   49.0 ms
    f99       153.67 us/call  inner=320    burst=   49.2 ms
    g102      139.54 us/call  inner=359    burst=   50.1 ms
    g27       188.88 us/call  inner=266    burst=   50.2 ms
    g85       139.07 us/call  inner=343    burst=   47.7 ms
    g86       114.21 us/call  inner=437    burst=   49.9 ms
    g89       123.67 us/call  inner=400    burst=   49.5 ms
    g98       143.57 us/call  inner=349    burst=   50.1 ms
    g99       128.58 us/call  inner=391    burst=   50.3 ms
    legA      124.94 us/call  inner=395    burst=   49.4 ms
    legB      488.63 us/call  inner=103    burst=   50.3 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 0.6044x  chain/s27 0.6478x
      s85  gs128      bank vs s86  1.2157x [1.2100-1.2257]  gemm/s27 0.7363x  chain/s27 0.7749x
      s89  gs256      bank vs s86  1.0838x [1.0762-1.1000]  gemm/s27 0.6548x  chain/s27 0.6999x
      s99  per-token                                         gemm/s27 0.6806x  chain/s27 0.7065x
      s98  gs128      bank vs s99  1.1171x [1.1149-1.1364]  gemm/s27 0.7605x  chain/s27 0.7911x
      s102 gs256      bank vs s99  1.0855x [1.0820-1.0946]  gemm/s27 0.7383x  chain/s27 0.7677x
      BEST grouped s89 vs BEST per-token s86: gemm 1.0838x [1.0762-1.1000]  chain 1.0806x [1.0599-1.0894]

  down_proj k=17408 n=5120 M=2048   burst>= 50 ms, reps=7
    f102     3289.54 us/call  inner=16     burst=   52.6 ms
    f27      1381.40 us/call  inner=37     burst=   51.1 ms
    f85      1758.84 us/call  inner=29     burst=   51.0 ms
    f86      1532.06 us/call  inner=33     burst=   50.6 ms
    f89      1686.56 us/call  inner=31     burst=   52.3 ms
    f98      3473.13 us/call  inner=15     burst=   52.1 ms
    f99      3112.51 us/call  inner=16     burst=   49.8 ms
    g102     2875.56 us/call  inner=18     burst=   51.8 ms
    g27       891.45 us/call  inner=61     burst=   54.4 ms
    g85      1246.93 us/call  inner=41     burst=   51.1 ms
    g86      1050.31 us/call  inner=50     burst=   52.5 ms
    g89      1169.36 us/call  inner=44     burst=   51.5 ms
    g98      3000.50 us/call  inner=17     burst=   51.0 ms
    g99      2661.98 us/call  inner=20     burst=   53.2 ms
    legA     4053.00 us/call  inner=14     burst=   56.7 ms
    legB     1640.22 us/call  inner=31     burst=   50.8 ms
    per-arm ratios, paired within a rep:
      s27  per-token                                         gemm/s27 1.0000x  chain/s27 1.0000x
      s86  per-token                                         gemm/s27 1.1763x  chain/s27 1.1209x
      s85  gs128      bank vs s86  1.2212x [1.1426-1.2439]  gemm/s27 1.3977x  chain/s27 1.2942x
      s89  gs256      bank vs s86  1.1128x [1.0960-1.1833]  gemm/s27 1.3408x  chain/s27 1.2299x
      s99  per-token                                         gemm/s27 3.0457x  chain/s27 2.2722x
      s98  gs128      bank vs s99  1.1165x [1.0570-1.1583]  gemm/s27 3.3684x  chain/s27 2.5141x
      s102 gs256      bank vs s99  1.0861x [1.0559-1.0918]  gemm/s27 3.2544x  chain/s27 2.3829x
      BEST grouped s89 vs BEST per-token s27: gemm 1.3408x [1.2165-1.4003]  chain 1.2299x [1.1871-1.2789]

[sampler] 1186 samples over 59.4 s of process life; 50.6 s of that was inside a timed block

[under load] sm clock median 2685 MHz (min 1995, max 2730)  power median 420 W  sw_power_cap 598/1011 samples  hw_slowdown 0/1011
[tenancy after ] 2339916, 3116 MiB   (this process is pid 60)
