============================================================
PLAN
============================================================
  fixtures            : 9  (add-method, constants-in-module, find-usages, hidden-bug-in-body, large-file-search, multi-file-rename, re-read-after-edit, rename, run-tests-fix-bug)
  runs per side per fx: 5
  total claude calls  : 90
  per-call cost cap   : $0.50 USD (--max-budget-usd)
  worst-case spend    : $45.00 USD
  typical per-call    : <$0.30  (so usually $22.50-$31.50 total)
  experiment          : none

============================================================
FIXTURE: add-method
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.2240 turns=4 wall=12.7s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/5] vanilla... ✓ calls=3 cost=$0.1235 turns=4 wall=12.0s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [3/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=13.1s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [4/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=12.9s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [5/5] vanilla... ✓ calls=3 cost=$0.1235 turns=4 wall=12.3s  (multiply_then_add(2,3,1)=7; delegates to self.add)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=5 cost=$0.2767 turns=6 wall=17.9s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/5] beaglelathe... ✓ calls=5 cost=$0.1710 turns=6 wall=16.3s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [3/5] beaglelathe... ✓ calls=5 cost=$0.1714 turns=6 wall=21.0s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [4/5] beaglelathe... ✓ calls=5 cost=$0.1742 turns=6 wall=18.3s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [5/5] beaglelathe... ✓ calls=5 cost=$0.1723 turns=6 wall=19.5s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
============================================================
FIXTURE: constants-in-module
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1215 turns=4 wall=11.9s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/5] vanilla... ✓ calls=3 cost=$0.1217 turns=4 wall=13.5s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [3/5] vanilla... ✓ calls=3 cost=$0.1216 turns=4 wall=11.5s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [4/5] vanilla... ✓ calls=3 cost=$0.1220 turns=4 wall=14.5s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [5/5] vanilla... ✓ calls=3 cost=$0.1215 turns=4 wall=19.7s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=3 cost=$0.1269 turns=4 wall=12.9s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/5] beaglelathe... ✓ calls=3 cost=$0.1283 turns=4 wall=12.4s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [3/5] beaglelathe... ✓ calls=3 cost=$0.1271 turns=4 wall=12.8s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [4/5] beaglelathe... ✓ calls=3 cost=$0.1269 turns=4 wall=25.8s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [5/5] beaglelathe... ✓ calls=3 cost=$0.1272 turns=4 wall=12.8s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
============================================================
FIXTURE: find-usages
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=19 cost=$0.4611 turns=20 wall=66.4s  (6 callers identified correctly)
    [2/5] vanilla... ✓ calls=20 cost=$0.4850 turns=21 wall=73.9s  (6 callers identified correctly)
    [3/5] vanilla... ✓ calls=21 cost=$0.3221 turns=22 wall=42.8s  (6 callers identified correctly)
    [4/5] vanilla... ✓ calls=20 cost=$0.3070 turns=21 wall=44.2s  (6 callers identified correctly)
    [5/5] vanilla... ✓ calls=6 cost=$0.2127 turns=7 wall=36.4s  (6 callers identified correctly)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1878 turns=5 wall=23.2s bl  (6 callers identified correctly)
    [2/5] beaglelathe... ✓ calls=7 cost=$0.2546 turns=8 wall=35.4s bl  (6 callers identified correctly)
    [3/5] beaglelathe... ✓ calls=6 cost=$0.2258 turns=7 wall=38.1s bl  (6 callers identified correctly)
    [4/5] beaglelathe... ✓ calls=6 cost=$0.2349 turns=7 wall=31.3s bl  (6 callers identified correctly)
    [5/5] beaglelathe... ✓ calls=5 cost=$0.2121 turns=6 wall=30.4s bl  (6 callers identified correctly)
============================================================
FIXTURE: hidden-bug-in-body
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1265 turns=4 wall=15.5s  (all 5 truth-table cases pass)
    [2/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=16.6s  (all 5 truth-table cases pass)
    [3/5] vanilla... ✓ calls=3 cost=$0.1235 turns=4 wall=12.4s  (all 5 truth-table cases pass)
    [4/5] vanilla... ✓ calls=3 cost=$0.1264 turns=4 wall=18.0s  (all 5 truth-table cases pass)
    [5/5] vanilla... ✓ calls=3 cost=$0.1263 turns=4 wall=13.0s  (all 5 truth-table cases pass)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1563 turns=5 wall=20.9s bl  (all 5 truth-table cases pass)
    [2/5] beaglelathe... ✓ calls=3 cost=$0.1274 turns=4 wall=9.9s bl  (all 5 truth-table cases pass)
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1483 turns=5 wall=16.9s bl  (all 5 truth-table cases pass)
    [4/5] beaglelathe... ✓ calls=3 cost=$0.1297 turns=4 wall=12.5s bl  (all 5 truth-table cases pass)
    [5/5] beaglelathe... ✓ calls=3 cost=$0.1269 turns=4 wall=9.8s bl  (all 5 truth-table cases pass)
============================================================
FIXTURE: large-file-search
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=4 cost=$0.1456 turns=5 wall=18.9s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/5] vanilla... ✓ calls=4 cost=$0.1455 turns=5 wall=29.5s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [3/5] vanilla... ✓ calls=4 cost=$0.1383 turns=5 wall=15.0s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [4/5] vanilla... ✓ calls=5 cost=$0.1652 turns=6 wall=19.7s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [5/5] vanilla... ✓ calls=4 cost=$0.1457 turns=5 wall=20.9s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1880 turns=5 wall=16.9s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/5] beaglelathe... ✓ calls=4 cost=$0.1930 turns=5 wall=21.6s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1582 turns=5 wall=18.6s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [4/5] beaglelathe... ✓ calls=4 cost=$0.1578 turns=5 wall=16.7s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [5/5] beaglelathe... ✓ calls=4 cost=$0.1900 turns=5 wall=29.0s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
============================================================
FIXTURE: multi-file-rename
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=27 cost=$0.4185 turns=28 wall=46.1s  (renamed 12/12 files)
    [2/5] vanilla... [claude exited 1]

✗ calls=22 cost=$0.5117 turns=22 wall=68.7s ⚠️  (renamed=8/12; leftover handleAuth in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/audit.ts']; handleLogin missing in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/audit.ts'])
    [3/5] vanilla... [claude exited 1]

✗ calls=22 cost=$0.5027 turns=22 wall=69.2s ⚠️  (renamed=7/12; leftover handleAuth in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts']; handleLogin missing in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts'])
    [4/5] vanilla... ✓ calls=27 cost=$0.4206 turns=28 wall=46.7s  (renamed 12/12 files)
    [5/5] vanilla... ✓ calls=27 cost=$0.4584 turns=28 wall=52.1s  (renamed 12/12 files)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=2 cost=$0.1158 turns=3 wall=7.2s bl  (renamed 12/12 files)
    [2/5] beaglelathe... ✓ calls=2 cost=$0.1158 turns=3 wall=7.8s bl  (renamed 12/12 files)
    [3/5] beaglelathe... ✓ calls=2 cost=$0.1158 turns=3 wall=7.7s bl  (renamed 12/12 files)
    [4/5] beaglelathe... ✓ calls=2 cost=$0.1157 turns=3 wall=7.9s bl  (renamed 12/12 files)
    [5/5] beaglelathe... ✓ calls=2 cost=$0.1157 turns=3 wall=7.5s bl  (renamed 12/12 files)
============================================================
FIXTURE: re-read-after-edit
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=5 cost=$0.1738 turns=6 wall=29.5s  (config edited; re-read confirmed DEBUG=True)
    [2/5] vanilla... ✓ calls=5 cost=$0.1643 turns=6 wall=19.0s  (config edited; re-read confirmed DEBUG=True)
    [3/5] vanilla... ✓ calls=5 cost=$0.1697 turns=6 wall=21.8s  (config edited; re-read confirmed DEBUG=True)
    [4/5] vanilla... ✓ calls=5 cost=$0.1677 turns=6 wall=21.6s  (config edited; re-read confirmed DEBUG=True)
    [5/5] vanilla... ✓ calls=5 cost=$0.1678 turns=6 wall=22.0s  (config edited; re-read confirmed DEBUG=True)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1479 turns=5 wall=15.9s bl  (config edited; re-read confirmed DEBUG=True)
    [2/5] beaglelathe... ✓ calls=4 cost=$0.1489 turns=5 wall=18.4s bl  (config edited; re-read confirmed DEBUG=True)
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1478 turns=5 wall=26.8s bl  (config edited; re-read confirmed DEBUG=True)
    [4/5] beaglelathe... ✓ calls=4 cost=$0.1489 turns=5 wall=19.6s bl  (config edited; re-read confirmed DEBUG=True)
    [5/5] beaglelathe... ✓ calls=4 cost=$0.1490 turns=5 wall=15.8s bl  (config edited; re-read confirmed DEBUG=True)
============================================================
FIXTURE: rename
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=9 cost=$0.2372 turns=10 wall=30.1s  (renamed 3/3 files)
    [2/5] vanilla... ✓ calls=5 cost=$0.1573 turns=6 wall=16.7s  (renamed 3/3 files)
    [3/5] vanilla... ✓ calls=11 cost=$0.2756 turns=12 wall=40.5s  (renamed 3/3 files)
    [4/5] vanilla... ✓ calls=5 cost=$0.1602 turns=6 wall=19.1s  (renamed 3/3 files)
    [5/5] vanilla... ✓ calls=4 cost=$0.1405 turns=5 wall=29.9s  (renamed 3/3 files)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=2 cost=$0.1068 turns=3 wall=7.5s bl  (renamed 3/3 files)
    [2/5] beaglelathe... ✓ calls=2 cost=$0.1066 turns=3 wall=7.8s bl  (renamed 3/3 files)
    [3/5] beaglelathe... ✓ calls=2 cost=$0.1068 turns=3 wall=19.7s bl  (renamed 3/3 files)
    [4/5] beaglelathe... ✓ calls=2 cost=$0.1067 turns=3 wall=7.0s bl  (renamed 3/3 files)
    [5/5] beaglelathe... ✓ calls=2 cost=$0.1066 turns=3 wall=7.2s bl  (renamed 3/3 files)
============================================================
FIXTURE: run-tests-fix-bug
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=5 cost=$0.1652 turns=6 wall=19.9s  (all 6 unittest tests pass; divide(10,2)=5)
    [2/5] vanilla... ✓ calls=7 cost=$0.1737 turns=8 wall=20.3s  (all 6 unittest tests pass; divide(10,2)=5)
    [3/5] vanilla... ✓ calls=5 cost=$0.1651 turns=6 wall=20.5s  (all 6 unittest tests pass; divide(10,2)=5)
    [4/5] vanilla... ✓ calls=5 cost=$0.1651 turns=6 wall=30.0s  (all 6 unittest tests pass; divide(10,2)=5)
    [5/5] vanilla... ✓ calls=5 cost=$0.1647 turns=6 wall=18.0s  (all 6 unittest tests pass; divide(10,2)=5)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=5 cost=$0.1726 turns=6 wall=27.0s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [2/5] beaglelathe... ✓ calls=8 cost=$0.2187 turns=9 wall=25.1s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [3/5] beaglelathe... ✓ calls=6 cost=$0.1943 turns=7 wall=20.5s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [4/5] beaglelathe... ✓ calls=8 cost=$0.2018 turns=9 wall=21.9s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [5/5] beaglelathe... ✓ calls=7 cost=$0.2081 turns=8 wall=27.6s bl  (all 6 unittest tests pass; divide(10,2)=5)

================================================================================================
PER-FIXTURE PASS RATE + STRUCTURAL DELTAS (means across passing runs only)
================================================================================================
  fixture                van pass  BL pass  BL-used   Δcalls   Δturns   Δoutput    Δcost
  --------------------------------------------------------------------------------------
  add-method               5/5      5/5      5/5      -66.7%   -50.0%    -63.3%   -34.4%
  constants-in-module      5/5      5/5      5/5       +0.0%    +0.0%     -1.2%    -4.6%
  find-usages              5/5      5/5      5/5      +67.4%   +63.7%    +43.5%   +37.6%
  hidden-bug-in-body       5/5      5/5      5/5      -13.3%   -10.0%     -3.7%    -9.9%
  large-file-search        5/5      5/5      5/5       +4.8%    +3.8%     -5.8%   -19.8%
  multi-file-rename        3/5      5/5      5/5      +92.6%   +89.3%    +93.1%   +73.2%
  re-read-after-edit       5/5      5/5      5/5      +20.0%   +16.7%    +39.2%   +12.0%
  rename                   5/5      5/5      5/5      +70.6%   +61.5%    +71.5%   +45.0%
  run-tests-fix-bug        5/5      5/5      5/5      -25.9%   -21.9%    -13.2%   -19.4%

================================================================================================
OVERALL (passing runs only, weighted equally per fixture)
================================================================================================
  pass rate     : vanilla 43/45  |  BL 45/45
  tool calls    : vanilla mean=7.4  |  BL mean=4.0
  turns         : vanilla mean=8.4  |  BL mean=5.0
  output tokens : vanilla mean=1,243  |  BL mean=720
  cost/run (actual)       : vanilla mean=$0.1960  |  BL mean=$0.1587
  cost/run (claude-sonnet-4-6 ): vanilla mean=$0.1176  |  BL mean=$0.0952
  cost/run (claude-opus-4-7   ): vanilla mean=$0.5879  |  BL mean=$0.4762
  wall time               : vanilla mean=25.1s  |  BL mean=17.9s
  BL tools used : 45/45 BL runs actually invoked a BL MCP tool (others fell back to vanilla)

  SAVINGS PER RUN (mean BL vs vanilla, positive = BL cheaper):
    actual                : $+0.0372 saved
    claude-sonnet-4-6     : $+0.0223 saved
    claude-opus-4-7       : $+0.1117 saved
  VERDICT: BL was cheaper than vanilla (mean cost lower across all priced models)

  raw event logs: /Users/thebeagle/dev/beagle-lathe-all/beagle-lathe/bench-logs/
