============================================================
PLAN
============================================================
  fixtures            : 9  (add-method, constants-in-module, find-usages, hidden-bug-in-body, large-file-search, multi-file-rename, re-read-after-edit, rename, run-tests-fix-bug)
  runs per side per fx: 2
  total claude calls  : 36
  per-call cost cap   : $0.50 USD (--max-budget-usd)
  worst-case spend    : $18.00 USD
  typical per-call    : <$0.30  (so usually $9.00-$12.60 total)
  experiment          : none

============================================================
FIXTURE: add-method
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=12.8s  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/2] vanilla... ✓ calls=3 cost=$0.1234 turns=4 wall=15.0s  (multiply_then_add(2,3,1)=7; delegates to self.add)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=5 cost=$0.1727 turns=6 wall=21.3s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
    [2/2] beaglelathe... ✓ calls=5 cost=$0.1741 turns=6 wall=17.8s bl  (multiply_then_add(2,3,1)=7; delegates to self.add)
============================================================
FIXTURE: constants-in-module
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1214 turns=4 wall=11.9s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/2] vanilla... ✓ calls=3 cost=$0.1212 turns=4 wall=12.4s  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=3 cost=$0.1268 turns=4 wall=14.0s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
    [2/2] beaglelathe... ✓ calls=3 cost=$0.1291 turns=4 wall=12.5s bl  (answer.txt=7 (matches MAX_RETRIES from src/limits.py))
============================================================
FIXTURE: find-usages
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=8 cost=$0.2607 turns=9 wall=37.0s  (6 callers identified correctly)
    [2/2] vanilla... ✓ calls=19 cost=$0.4633 turns=20 wall=61.7s  (6 callers identified correctly)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=7 cost=$0.2636 turns=8 wall=37.4s bl  (6 callers identified correctly)
    [2/2] beaglelathe... ✓ calls=7 cost=$0.2539 turns=8 wall=47.4s bl  (6 callers identified correctly)
============================================================
FIXTURE: hidden-bug-in-body
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=3 cost=$0.1260 turns=4 wall=13.6s  (all 5 truth-table cases pass)
    [2/2] vanilla... ✓ calls=3 cost=$0.1237 turns=4 wall=12.5s  (all 5 truth-table cases pass)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.1520 turns=5 wall=14.0s bl  (all 5 truth-table cases pass)
    [2/2] beaglelathe... ✓ calls=4 cost=$0.1515 turns=5 wall=14.9s bl  (all 5 truth-table cases pass)
============================================================
FIXTURE: large-file-search
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=4 cost=$0.1461 turns=5 wall=19.2s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/2] vanilla... ✓ calls=4 cost=$0.1448 turns=5 wall=17.5s  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.1988 turns=5 wall=25.5s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
    [2/2] beaglelathe... ✓ calls=4 cost=$0.1898 turns=5 wall=19.1s bl  (answer.txt=OrderProcessor (the class whose handle() returns 'authoritative'))
============================================================
FIXTURE: multi-file-rename
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=27 cost=$0.4168 turns=28 wall=53.0s  (renamed 12/12 files)
    [2/2] vanilla... ✓ calls=27 cost=$0.4543 turns=28 wall=50.4s  (renamed 12/12 files)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=2 cost=$0.1157 turns=3 wall=7.6s bl  (renamed 12/12 files)
    [2/2] beaglelathe... ✓ calls=2 cost=$0.1156 turns=3 wall=7.0s bl  (renamed 12/12 files)
============================================================
FIXTURE: re-read-after-edit
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=5 cost=$0.1634 turns=6 wall=18.7s  (config edited; re-read confirmed DEBUG=True)
    [2/2] vanilla... ✓ calls=5 cost=$0.1671 turns=6 wall=21.0s  (config edited; re-read confirmed DEBUG=True)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=4 cost=$0.1478 turns=5 wall=14.7s bl  (config edited; re-read confirmed DEBUG=True)
    [2/2] beaglelathe... ✓ calls=4 cost=$0.1479 turns=5 wall=16.3s bl  (config edited; re-read confirmed DEBUG=True)
============================================================
FIXTURE: rename
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=4 cost=$0.1430 turns=5 wall=16.7s  (renamed 3/3 files)
    [2/2] vanilla... ✓ calls=4 cost=$0.1408 turns=5 wall=16.1s  (renamed 3/3 files)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=2 cost=$0.1068 turns=3 wall=8.2s bl  (renamed 3/3 files)
    [2/2] beaglelathe... ✓ calls=2 cost=$0.1077 turns=3 wall=7.8s bl  (renamed 3/3 files)
============================================================
FIXTURE: run-tests-fix-bug
============================================================
  -- vanilla --
    [1/2] vanilla... ✓ calls=5 cost=$0.1642 turns=6 wall=21.5s  (all 6 unittest tests pass; divide(10,2)=5)
    [2/2] vanilla... ✓ calls=5 cost=$0.1646 turns=6 wall=20.9s  (all 6 unittest tests pass; divide(10,2)=5)
  -- beaglelathe --
    [1/2] beaglelathe... ✓ calls=7 cost=$0.1946 turns=8 wall=17.8s bl  (all 6 unittest tests pass; divide(10,2)=5)
    [2/2] beaglelathe... ✓ calls=5 cost=$0.1721 turns=6 wall=198.8s bl  (all 6 unittest tests pass; divide(10,2)=5)

================================================================================================
PER-FIXTURE PASS RATE + STRUCTURAL DELTAS (means across passing runs only)
================================================================================================
  fixture                van pass  BL pass  BL-used   Δcalls   Δturns   Δoutput    Δcost
  --------------------------------------------------------------------------------------
  add-method               2/2      2/2      2/2      -66.7%   -50.0%    -75.6%   -40.4%
  constants-in-module      2/2      2/2      2/2       +0.0%    +0.0%     -0.2%    -5.4%
  find-usages              2/2      2/2      2/2      +48.1%   +44.8%    +31.1%   +28.5%
  hidden-bug-in-body       2/2      2/2      2/2      -33.3%   -25.0%    -33.0%   -21.5%
  large-file-search        2/2      2/2      2/2       +0.0%    +0.0%     -7.6%   -33.6%
  multi-file-rename        2/2      2/2      2/2      +92.6%   +89.3%    +93.0%   +73.4%
  re-read-after-edit       2/2      2/2      2/2      +20.0%   +16.7%    +34.3%   +10.5%
  rename                   2/2      2/2      2/2      +50.0%   +40.0%    +52.6%   +24.4%
  run-tests-fix-bug        2/2      2/2      2/2      -20.0%   -16.7%     +2.4%   -11.5%

================================================================================================
OVERALL (passing runs only, weighted equally per fixture)
================================================================================================
  pass rate     : vanilla 18/18  |  BL 18/18
  tool calls    : vanilla mean=7.5  |  BL mean=4.1
  turns         : vanilla mean=8.5  |  BL mean=5.1
  output tokens : vanilla mean=1,293  |  BL mean=751
  cost/run (actual)       : vanilla mean=$0.1982  |  BL mean=$0.1623
  cost/run (claude-sonnet-4-6 ): vanilla mean=$0.1189  |  BL mean=$0.0974
  cost/run (claude-opus-4-7   ): vanilla mean=$0.5947  |  BL mean=$0.4868
  wall time               : vanilla mean=24.0s  |  BL mean=27.9s
  BL tools used : 18/18 BL runs actually invoked a BL MCP tool (others fell back to vanilla)

  SAVINGS PER RUN (mean BL vs vanilla, positive = BL cheaper):
    actual                : $+0.0360 saved
    claude-sonnet-4-6     : $+0.0216 saved
    claude-opus-4-7       : $+0.1080 saved
  VERDICT: BL was cheaper than vanilla (mean cost lower across all priced models)

  raw event logs: /Users/thebeagle/dev/beagle-lathe-all/beagle-lathe/bench-logs/
