============================================================
PLAN
============================================================
  fixtures            : 9  (add-method, constants-in-module, find-usages, hidden-bug-in-body, large-file-search, multi-file-rename, re-read-after-edit, rename, run-tests-fix-bug)
  runs per side per fx: 2
  total claude calls  : 36
  per-call cost cap   : $0.50 USD (--max-budget-usd)
  worst-case spend    : $18.00 USD
  typical per-call    : <$0.30  (so usually $9.00-$12.60 total)
  experiment          : none

============================================================
FIXTURE: add-method
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1249 turns=4 wall=15.4s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/2] vanilla... ✓ calls=3 cost=$0.1249 turns=4 wall=14.2s  (multiply_then_add(2,3,1)=7; delegates to self.add)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=3 cost=$0.1795 turns=4 wall=8.8s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/2] beaglelathe... ✓ calls=3 cost=$0.1794 turns=4 wall=9.4s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
============================================================
FIXTURE: constants-in-module
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1226 turns=4 wall=11.7s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/2] vanilla... ✓ calls=3 cost=$0.1228 turns=4 wall=13.4s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=3 cost=$0.1795 turns=4 wall=17.5s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/2] beaglelathe... ✓ calls=3 cost=$0.1820 turns=4 wall=14.0s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
============================================================
FIXTURE: find-usages
============================================================
  -- vanilla --
    [1/2] vanilla... [claude exited 1]

✓ calls=21 cost=$0.5212 turns=22 wall=75.2s ⚠️  (6 callers identified correctly)
    [2/2] vanilla... ✓ calls=19 cost=$0.4811 turns=20 wall=67.7s  (6 callers identified correctly)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.2403 turns=5 wall=50.7s bl  (6 callers identified correctly)
    [2/2] beaglelathe... ✓ calls=7 cost=$0.2700 turns=8 wall=33.3s bl  (6 callers identified correctly)
============================================================
FIXTURE: hidden-bug-in-body
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1249 turns=4 wall=13.7s  (all 5 truth-table cases pass)
    [2/2] vanilla... ✓ calls=3 cost=$0.1278 turns=4 wall=16.7s  (all 5 truth-table cases pass)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=3 cost=$0.1789 turns=4 wall=12.2s bl  (all 5 truth-table cases pass)
    [2/2] beaglelathe... ✓ calls=3 cost=$0.1801 turns=4 wall=11.1s bl  (all 5 truth-table cases pass)
============================================================
FIXTURE: large-file-search
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=4 cost=$0.1479 turns=5 wall=22.3s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/2] vanilla... ✓ calls=4 cost=$0.1426 turns=5 wall=16.4s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.2127 turns=5 wall=17.4s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/2] beaglelathe... ✓ calls=4 cost=$0.2344 turns=5 wall=18.3s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
============================================================
FIXTURE: multi-file-rename
============================================================
  -- vanilla --
    [1/2] vanilla... [claude exited 1]

✓ calls=29 cost=$0.5473 turns=29 wall=57.6s ⚠️  (renamed 12/12 files)
    [2/2] vanilla... [claude exited 1]

✗ calls=22 cost=$0.5065 turns=22 wall=70.7s ⚠️  (renamed=7/12; leftover handleAuth in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts']; handleLogin missing in: ['src/api/users.ts', 'src/api/posts.ts', 'src/api/comments.ts', 'src/admin/dashboard.ts', 'src/admin/audit.ts'])
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=2 cost=$0.1670 turns=3 wall=9.5s bl  (renamed 12/12 files)
    [2/2] beaglelathe... ✓ calls=2 cost=$0.1675 turns=3 wall=8.5s bl  (renamed 12/12 files)
============================================================
FIXTURE: re-read-after-edit
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=5 cost=$0.1727 turns=6 wall=22.8s  (config edited; re-read confirmed DEBUG=True)
    [2/2] vanilla... ✓ calls=5 cost=$0.1676 turns=6 wall=22.4s  (config edited; re-read confirmed DEBUG=True)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.2013 turns=5 wall=29.8s bl  (config edited; re-read confirmed DEBUG=True)
    [2/2] beaglelathe... ✓ calls=4 cost=$0.2047 turns=5 wall=16.0s bl  (config edited; re-read confirmed DEBUG=True)
============================================================
FIXTURE: rename
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=9 cost=$0.3788 turns=10 wall=33.9s  (renamed 3/3 files)
    [2/2] vanilla... ✓ calls=9 cost=$0.2849 turns=10 wall=32.3s  (renamed 3/3 files)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=2 cost=$0.1658 turns=3 wall=9.6s bl  (renamed 3/3 files)
    [2/2] beaglelathe... ✓ calls=2 cost=$0.1632 turns=3 wall=7.5s bl  (renamed 3/3 files)
============================================================
FIXTURE: run-tests-fix-bug
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=5 cost=$0.1659 turns=6 wall=23.8s  (all 6 unittest tests pass; divide(10,2)=5)
    [2/2] vanilla... ✓ calls=5 cost=$0.1519 turns=6 wall=18.2s  (all 6 unittest tests pass; divide(10,2)=5)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=7 cost=$0.2389 turns=8 wall=22.3s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [2/2] beaglelathe... ✓ calls=7 cost=$0.2356 turns=8 wall=20.4s bl  (all 6 unittest tests pass; divide(10,2)=5)

================================================================================================
PER-FIXTURE PASS RATE + STRUCTURAL DELTAS (means across passing runs only)
================================================================================================
  fixture                van pass  BL pass  BL-used   Δcalls   Δturns   Δoutput    Δcost
  --------------------------------------------------------------------------------------
  add-method               2/2      2/2      2/2       +0.0%    +0.0%    +25.8%   -43.7%
  constants-in-module      2/2      2/2      2/2       +0.0%    +0.0%     +9.8%   -47.3%
  find-usages              2/2      2/2      2/2      +72.5%   +69.0%    +54.6%   +49.1%
  hidden-bug-in-body       2/2      2/2      2/2       +0.0%    +0.0%    +31.4%   -42.1%
  large-file-search        2/2      2/2      2/2       +0.0%    +0.0%    -14.1%   -53.9%
  multi-file-rename        1/2      2/2      2/2      +93.1%   +89.7%    +84.5%   +69.4%
  re-read-after-edit       2/2      2/2      2/2      +20.0%   +16.7%    +40.4%   -19.3%
  rename                   2/2      2/2      2/2      +77.8%   +70.0%    +74.6%   +50.4%
  run-tests-fix-bug        2/2      2/2      2/2      -40.0%   -33.3%    -31.7%   -49.3%

================================================================================================
OVERALL (passing runs only, weighted equally per fixture)
================================================================================================
  pass rate     : vanilla 17/18  |  BL 18/18
  tool calls    : vanilla mean=7.8  |  BL mean=3.7
  turns         : vanilla mean=8.8  |  BL mean=4.7
  output tokens : vanilla mean=1,241  |  BL mean=691
  cost/run (actual)       : vanilla mean=$0.2300  |  BL mean=$0.1989
  cost/run (claude-sonnet-4-6 ): vanilla mean=$0.1295  |  BL mean=$0.1194
  cost/run (claude-opus-4-7   ): vanilla mean=$0.6477  |  BL mean=$0.5968
  wall time               : vanilla mean=28.1s  |  BL mean=17.6s
  BL tools used : 18/18 BL runs actually invoked a BL MCP tool (others fell back to vanilla)

  SAVINGS PER RUN (mean BL vs vanilla, positive = BL cheaper):
    actual                : $+0.0311 saved
    claude-sonnet-4-6     : $+0.0102 saved
    claude-opus-4-7       : $+0.0509 saved
  VERDICT: BL was cheaper than vanilla (mean cost lower across all priced models)

  raw event logs: /Users/thebeagle/dev/beagle-lathe-all/beagle-lathe/bench-logs/
