# Promptbench

> PromptBench enables developers to compare different prompt compositions by running Cartesian products of system templates, user payloads, and skills across multiple LLM providers. Its core goal is to evaluate how variations in prompt structure affect model output, supporting iterative prompt engineering. The tool also facilitates reproducible benchmarking through atomic artifact capture and structured logging, with optional TUI for interactive exploration and advanced configuration.

**Remember:**
- Triple-Layer Recursive Fusion (House, Influence, Target)
- Composition Modes: Engine, Marker, Append
- Multi-Provider Support (LM Studio, Codex CLI, Gemini CLI)
- Atomic Artifact Capture with structured output files
- Skill Router for auto-mapping skills based on template/payload content
- Event Stream (JSONL) for job lifecycle tracking

## Docs
- [Prompt Bench Overview.Prd](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/docs/Prompt-bench Overview.prd.md): install & quickstart.
- [Prompt Bench Prd](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/docs/prompt-bench_PRD.md): docs page.
- [Promptbench Tui.Prd](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/docs/promptbench-tui.prd.md): docs page.
- [Bugs](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/bugs.md): docs page.
- [Lm Studio](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/lm-studio.md): docs page.
- [Mcp](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/mcp.md): docs page.
- [Misc](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/misc.md): docs page.
- [Raw Exports](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/raw-exports.md): docs page.
- [Standardize Promptbench Workspace](https://github.com/AcidicSoil/promptbench/blob/main/docs/oracle-questions-2026-01-09-154207-run-2026-01-09-154207/packs/standardize-promptbench-workspace.md): docs page.

## Tutorials
- [Expected Output](https://github.com/AcidicSoil/promptbench/blob/main/examples/expected-output.md): worked example.
- [User Input](https://github.com/AcidicSoil/promptbench/blob/main/examples/user-input.md): worked example.
- [Example Prd](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/templates/example_prd.txt): worked example.
- [Example Prd Rpg](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/templates/example_prd_rpg.txt): worked example.
- [Composed System](https://github.com/AcidicSoil/promptbench/blob/main/benchmarks/demo/71d38850b64a/composed_system.txt): worked example.
- [Output](https://github.com/AcidicSoil/promptbench/blob/main/benchmarks/demo/71d38850b64a/output.txt): worked example.

## API
- [Lm Studio Api Error](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/raw-exports/LM Studio API Error.md): docs page.

## Concepts
- [Composition Modes Redesign](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Composition Modes Redesign.md): docs page.
- [Promptbench Mcp Server Design Implementation](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/mcp/promptbench_mcp_server_design_implementation.md): docs page.

## Optional
- [Agents](https://github.com/AcidicSoil/promptbench/blob/main/.taskmaster/AGENTS.md): docs page.
- [README](https://github.com/AcidicSoil/promptbench/blob/main/README.md): docs page.
- [Skill Context Router](https://github.com/AcidicSoil/promptbench/blob/main/scripts/skill_context_router.py): docs page.
- [When To Use This Skill](https://github.com/AcidicSoil/promptbench/blob/main/skills/when-to-use-this-skill.md): docs page.

## .Tickets
- [Progress View Issue](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Progress View Issue.md): docs page.
- [Prompt Bench Pipeline Enhancement](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Prompt-Bench Pipeline Enhancement.md): docs page.
- [Promptbench Composition Modes](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Promptbench Composition Modes.md): docs page.
- [Promptbench Lm Studio Integration](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/PromptBench LM Studio Integration.md): docs page.
- [Skill Router Improvement Plan](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Skill Router Improvement Plan.md): docs page.
- [Web Search For Promptbench Skills](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/Web-search for promptbench skills.md): docs page.
- [Expose Mcp Server For Promptbench](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/mcp/Expose MCP Server for Promptbench.md): docs page.
- [Expose Promptbench As Mcp](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/mcp/Expose promptbench as MCP.md): docs page.
- [Lm Studio Model Selection](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/lm-studio/LM-Studio-Model-Selection.md): docs page.
- [Lm Studio Sampling Params](https://github.com/AcidicSoil/promptbench/blob/main/.tickets/lm-studio/LM Studio Sampling Params.md): docs page.

## Payloads
- [Haiku](https://github.com/AcidicSoil/promptbench/blob/main/payloads/haiku.md): docs page.
- [Transcript Cleaner V2](https://github.com/AcidicSoil/promptbench/blob/main/payloads/transcript-cleaner_v2.md): docs page.

## Promptbench
- [Cli](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/cli.py): docs page.
- [Init](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/__init__.py): docs page.
- [Main](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/__main__.py): docs page.
- [Base](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/providers/base.py): docs page.
- [Codex Cli](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/providers/codex_cli.py): docs page.
- [Compose](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/core/compose.py): docs page.
- [Config](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/core/config.py): docs page.
- [Constants](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/core/constants.py): docs page.
- [Contract](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/core/contract.py): docs page.
- [Discovery](https://github.com/AcidicSoil/promptbench/blob/main/promptbench/core/discovery.py): docs page.

## Templates
- [Creative Writer](https://github.com/AcidicSoil/promptbench/blob/main/templates/creative_writer.md): docs page.
- [Prompt Composition Engine](https://github.com/AcidicSoil/promptbench/blob/main/templates/prompt-composition-engine.md): docs page.
- [Prompt Composition Engine V3](https://github.com/AcidicSoil/promptbench/blob/main/templates/prompt-composition-engine_v3.md): docs page.

## Tests
- [Test Artifacts](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_artifacts.py): docs page.
- [Test Artifacts Concurrency](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_artifacts_concurrency.py): docs page.
- [Test Cli](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_cli.py): docs page.
- [Test Cli Providers](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_cli_providers.py): docs page.
- [Test Compose](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_compose.py): docs page.
- [Test Config](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_config.py): docs page.
- [Test Contract](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_contract.py): docs page.
- [Test Core](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_core.py): docs page.
- [Test Engine Composition](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_engine_composition.py): docs page.
- [Test Eventing Executor](https://github.com/AcidicSoil/promptbench/blob/main/tests/test_eventing_executor.py): docs page.
