============================================================
PLAN
============================================================
  fixtures            : 9  (add-method, constants-in-module, find-usages, hidden-bug-in-body, large-file-search, multi-file-rename, re-read-after-edit, rename, run-tests-fix-bug)
  runs per side per fx: 5
  total claude calls  : 90
  per-call cost cap   : $0.50 USD (--max-budget-usd)
  worst-case spend    : $45.00 USD
  typical per-call    : <$0.30  (so usually $22.50-$31.50 total)
  experiment          : none

============================================================
FIXTURE: add-method
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=12.2s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=12.3s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [3/5] vanilla... ✓ calls=3 cost=$0.1236 turns=4 wall=12.8s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [4/5] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=19.2s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [5/5] vanilla... ✓ calls=3 cost=$0.1234 turns=4 wall=13.9s  (multiply_then_add(2,3,1)=7; delegates to self.add)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=5 cost=$0.2817 turns=6 wall=15.7s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/5] beaglelathe... ✓ calls=5 cost=$0.1783 turns=6 wall=20.4s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [3/5] beaglelathe... ✓ calls=5 cost=$0.1816 turns=6 wall=23.7s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [4/5] beaglelathe... ✓ calls=5 cost=$0.1848 turns=6 wall=21.3s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [5/5] beaglelathe... ✓ calls=5 cost=$0.1803 turns=6 wall=16.7s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
============================================================
FIXTURE: constants-in-module
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1214 turns=4 wall=11.5s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/5] vanilla... ✓ calls=3 cost=$0.1216 turns=4 wall=13.1s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [3/5] vanilla... ✓ calls=3 cost=$0.1215 turns=4 wall=12.0s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [4/5] vanilla... ✓ calls=3 cost=$0.1216 turns=4 wall=12.0s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [5/5] vanilla... ✓ calls=3 cost=$0.1213 turns=4 wall=12.2s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=3 cost=$0.1312 turns=4 wall=13.5s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/5] beaglelathe... ✓ calls=3 cost=$0.1312 turns=4 wall=12.7s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [3/5] beaglelathe... ✓ calls=3 cost=$0.1328 turns=4 wall=13.1s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [4/5] beaglelathe... ✓ calls=3 cost=$0.1319 turns=4 wall=13.1s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [5/5] beaglelathe... ✓ calls=3 cost=$0.1313 turns=4 wall=11.9s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
============================================================
FIXTURE: find-usages
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=19 cost=$0.4625 turns=20 wall=134.9s  (6 callers identified correctly)
    [2/5] vanilla... ✓ calls=20 cost=$0.4988 turns=21 wall=101.4s  (6 callers identified correctly)
    [3/5] vanilla... ✓ calls=6 cost=$0.2168 turns=7 wall=30.5s  (6 callers identified correctly)
    [4/5] vanilla... ✓ calls=5 cost=$0.2056 turns=6 wall=28.7s  (6 callers identified correctly)
    [5/5] vanilla... ✓ calls=5 cost=$0.2021 turns=6 wall=26.8s  (6 callers identified correctly)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=8 cost=$0.2764 turns=9 wall=34.2s bl  (6 callers identified correctly)
    [2/5] beaglelathe... ✓ calls=8 cost=$0.2822 turns=9 wall=34.2s bl  (6 callers identified correctly)
    [3/5] beaglelathe... ✓ calls=8 cost=$0.2732 turns=9 wall=33.2s bl  (6 callers identified correctly)
    [4/5] beaglelathe... ✓ calls=5 cost=$0.2216 turns=6 wall=29.2s bl  (6 callers identified correctly)
    [5/5] beaglelathe... ✓ calls=7 cost=$0.2542 turns=8 wall=31.3s bl  (6 callers identified correctly)
============================================================
FIXTURE: hidden-bug-in-body
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1262 turns=4 wall=14.0s  (all 5 truth-table cases pass)
    [2/5] vanilla... ✓ calls=3 cost=$0.1264 turns=4 wall=29.8s  (all 5 truth-table cases pass)
    [3/5] vanilla... ✓ calls=3 cost=$0.1262 turns=4 wall=13.7s  (all 5 truth-table cases pass)
    [4/5] vanilla... ✓ calls=3 cost=$0.1235 turns=4 wall=11.4s  (all 5 truth-table cases pass)
    [5/5] vanilla... ✓ calls=3 cost=$0.1262 turns=4 wall=13.7s  (all 5 truth-table cases pass)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1521 turns=5 wall=12.6s bl  (all 5 truth-table cases pass)
    [2/5] beaglelathe... ✓ calls=4 cost=$0.1530 turns=5 wall=16.3s bl  (all 5 truth-table cases pass)
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1617 turns=5 wall=17.2s bl  (all 5 truth-table cases pass)
    [4/5] beaglelathe... ✓ calls=4 cost=$0.1550 turns=5 wall=28.4s bl  (all 5 truth-table cases pass)
    [5/5] beaglelathe... ✓ calls=4 cost=$0.1568 turns=5 wall=15.5s bl  (all 5 truth-table cases pass)
============================================================
FIXTURE: large-file-search
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=4 cost=$0.1420 turns=5 wall=15.2s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/5] vanilla... ✓ calls=4 cost=$0.1419 turns=5 wall=15.8s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [3/5] vanilla... ✓ calls=5 cost=$0.1673 turns=6 wall=24.1s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [4/5] vanilla... ✓ calls=4 cost=$0.1462 turns=5 wall=18.4s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [5/5] vanilla... ✓ calls=4 cost=$0.1488 turns=5 wall=26.4s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=5 cost=$0.2122 turns=6 wall=19.1s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/5] beaglelathe... ✓ calls=4 cost=$0.2004 turns=5 wall=19.2s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1994 turns=5 wall=20.6s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [4/5] beaglelathe... ✓ calls=4 cost=$0.1999 turns=5 wall=20.7s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [5/5] beaglelathe... ✓ calls=4 cost=$0.2022 turns=5 wall=19.5s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
============================================================
FIXTURE: multi-file-rename
============================================================
  -- vanilla --
    [1/5] vanilla... [claude exited 1]

✗ calls=22 cost=$0.5011 turns=22 wall=73.0s ⚠️  (renamed=7/12; leftover handleAuth in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts']; handleLogin missing in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts'])
    [2/5] vanilla... ✓ calls=28 cost=$0.4426 turns=29 wall=51.7s  (renamed 12/12 files)
    [3/5] vanilla... ✓ calls=27 cost=$0.4519 turns=28 wall=48.0s  (renamed 12/12 files)
    [4/5] vanilla... ✓ calls=27 cost=$0.4540 turns=28 wall=49.2s  (renamed 12/12 files)
    [5/5] vanilla... [claude exited 1]

✗ calls=22 cost=$0.5036 turns=22 wall=75.8s ⚠️  (renamed=7/12; leftover handleAuth in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/middleware/guard.ts', 'src/middleware/logger.ts']; handleLogin missing in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/middleware/guard.ts', 'src/middleware/logger.ts'])
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=2 cost=$0.1195 turns=3 wall=8.2s bl  (renamed 12/12 files)
    [2/5] beaglelathe... ✓ calls=2 cost=$0.1195 turns=3 wall=23.7s bl  (renamed 12/12 files)
    [3/5] beaglelathe... ✓ calls=2 cost=$0.1197 turns=3 wall=18.8s bl  (renamed 12/12 files)
    [4/5] beaglelathe... ✓ calls=2 cost=$0.1212 turns=3 wall=8.8s bl  (renamed 12/12 files)
    [5/5] beaglelathe... ✓ calls=2 cost=$0.1196 turns=3 wall=23.9s bl  (renamed 12/12 files)
============================================================
FIXTURE: re-read-after-edit
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=5 cost=$0.1671 turns=6 wall=21.1s  (config edited; re-read confirmed DEBUG=True)
    [2/5] vanilla... ✓ calls=5 cost=$0.1675 turns=6 wall=23.5s  (config edited; re-read confirmed DEBUG=True)
    [3/5] vanilla... ✓ calls=5 cost=$0.1670 turns=6 wall=21.7s  (config edited; re-read confirmed DEBUG=True)
    [4/5] vanilla... ✓ calls=4 cost=$0.1445 turns=5 wall=18.8s  (config edited; re-read confirmed DEBUG=True)
    [5/5] vanilla... ✓ calls=5 cost=$0.1665 turns=6 wall=20.9s  (config edited; re-read confirmed DEBUG=True)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=4 cost=$0.1530 turns=5 wall=17.6s bl  (config edited; re-read confirmed DEBUG=True)
    [2/5] beaglelathe... ✓ calls=4 cost=$0.1540 turns=5 wall=22.2s bl  (config edited; re-read confirmed DEBUG=True)
    [3/5] beaglelathe... ✓ calls=4 cost=$0.1528 turns=5 wall=17.2s bl  (config edited; re-read confirmed DEBUG=True)
    [4/5] beaglelathe... ✓ calls=4 cost=$0.1534 turns=5 wall=14.9s bl  (config edited; re-read confirmed DEBUG=True)
    [5/5] beaglelathe... ✓ calls=4 cost=$0.1537 turns=5 wall=15.5s bl  (config edited; re-read confirmed DEBUG=True)
============================================================
FIXTURE: rename
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=3 cost=$0.1231 turns=4 wall=14.3s  (renamed 3/3 files)
    [2/5] vanilla... ✓ calls=4 cost=$0.1375 turns=5 wall=25.0s  (renamed 3/3 files)
    [3/5] vanilla... ✓ calls=3 cost=$0.1211 turns=4 wall=13.6s  (renamed 3/3 files)
    [4/5] vanilla... ✓ calls=5 cost=$0.1537 turns=6 wall=15.1s  (renamed 3/3 files)
    [5/5] vanilla... ✓ calls=4 cost=$0.1386 turns=5 wall=13.0s  (renamed 3/3 files)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=2 cost=$0.1105 turns=3 wall=8.9s bl  (renamed 3/3 files)
    [2/5] beaglelathe... ✓ calls=2 cost=$0.1116 turns=3 wall=7.9s bl  (renamed 3/3 files)
    [3/5] beaglelathe... ✓ calls=2 cost=$0.1105 turns=3 wall=7.9s bl  (renamed 3/3 files)
    [4/5] beaglelathe... ✓ calls=2 cost=$0.1105 turns=3 wall=8.2s bl  (renamed 3/3 files)
    [5/5] beaglelathe... ✓ calls=2 cost=$0.1105 turns=3 wall=7.3s bl  (renamed 3/3 files)
============================================================
FIXTURE: run-tests-fix-bug
============================================================
  -- vanilla --
    [1/5] vanilla... ✓ calls=5 cost=$0.1647 turns=6 wall=18.4s  (all 6 unittest tests pass; divide(10,2)=5)
    [2/5] vanilla... ✓ calls=5 cost=$0.1653 turns=6 wall=18.1s  (all 6 unittest tests pass; divide(10,2)=5)
    [3/5] vanilla... ✓ calls=5 cost=$0.1648 turns=6 wall=17.0s  (all 6 unittest tests pass; divide(10,2)=5)
    [4/5] vanilla... ✓ calls=5 cost=$0.1658 turns=6 wall=18.2s  (all 6 unittest tests pass; divide(10,2)=5)
    [5/5] vanilla... ✓ calls=5 cost=$0.1653 turns=6 wall=18.8s  (all 6 unittest tests pass; divide(10,2)=5)
  -- beaglelathe --
    [1/5] beaglelathe... ✓ calls=8 cost=$0.2226 turns=9 wall=23.9s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [2/5] beaglelathe... ✓ calls=7 cost=$0.2047 turns=8 wall=18.8s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [3/5] beaglelathe... ✓ calls=6 cost=$0.1964 turns=7 wall=18.3s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [4/5] beaglelathe... ✓ calls=6 cost=$0.2123 turns=7 wall=19.5s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [5/5] beaglelathe... ✓ calls=6 cost=$0.2030 turns=7 wall=18.6s bl  (all 6 unittest tests pass; divide(10,2)=5)

================================================================================================
PER-FIXTURE PASS RATE + STRUCTURAL DELTAS (means across passing runs only)
================================================================================================
  fixture                van pass  BL pass  BL-used   Δcalls   Δturns   Δoutput    Δcost
  --------------------------------------------------------------------------------------
  add-method               5/5      5/5      5/5      -66.7%   -50.0%    -77.6%   -62.9%
  constants-in-module      5/5      5/5      5/5       +0.0%    +0.0%     -1.2%    -8.4%
  find-usages              5/5      5/5      5/5      +34.5%   +31.7%    +20.4%   +17.5%
  hidden-bug-in-body       5/5      5/5      5/5      -33.3%   -25.0%    -21.2%   -23.9%
  large-file-search        5/5      5/5      5/5       +0.0%    +0.0%    -12.4%   -35.9%
  multi-file-rename        3/5      5/5      5/5      +92.7%   +89.4%    +93.0%   +73.3%
  re-read-after-edit       5/5      5/5      5/5      +16.7%   +13.8%    +31.8%    +5.6%
  rename                   5/5      5/5      5/5      +47.4%   +37.5%    +45.7%   +17.9%
  run-tests-fix-bug        5/5      5/5      5/5      -32.0%   -26.7%    -10.9%   -25.8%

================================================================================================
OVERALL (passing runs only, weighted equally per fixture)
================================================================================================
  pass rate     : vanilla 43/45  |  BL 45/45
  tool calls    : vanilla mean=6.3  |  BL mean=4.2
  turns         : vanilla mean=7.3  |  BL mean=5.2
  output tokens : vanilla mean=1,116  |  BL mean=773
  cost/run (actual)       : vanilla mean=$0.1825  |  BL mean=$0.1716
  cost/run (claude-sonnet-4-6 ): vanilla mean=$0.1095  |  BL mean=$0.1030
  cost/run (claude-opus-4-7   ): vanilla mean=$0.5475  |  BL mean=$0.5149
  wall time               : vanilla mean=24.7s  |  BL mean=18.3s
  BL tools used : 45/45 BL runs actually invoked a BL MCP tool (others fell back to vanilla)

  SAVINGS PER RUN (mean BL vs vanilla, positive = BL cheaper):
    actual                : $+0.0108 saved
    claude-sonnet-4-6     : $+0.0065 saved
    claude-opus-4-7       : $+0.0325 saved
  VERDICT: BL was cheaper than vanilla (mean cost lower across all priced models)

  raw event logs: /Users/thebeagle/dev/beagle-lathe-all/beagle-lathe/bench-logs/
