Metadata-Version: 2.4
Name: prompt-baker
Version: 0.1.0
Summary: Genetic optimization for prompt and model combinations on CSV benchmarks.
Project-URL: Homepage, https://github.com/sankhoroy/prompt-baker
Project-URL: Repository, https://github.com/sankhoroy/prompt-baker
Project-URL: Issues, https://github.com/sankhoroy/prompt-baker/issues
Author-email: sankhoroy <sankhoroy@gmail.com>
License: MIT
License-File: LICENSE
Keywords: classification,genetic-algorithm,llm,machine-learning,prompt-engineering,prompt-optimization,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: matplotlib>=3.9.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: rouge-score>=0.1.2
Requires-Dist: scikit-learn>=1.5.0
Requires-Dist: tqdm>=4.66.0
Description-Content-Type: text/markdown

<!-- <p align="center">
  <img src="assets/prompt-baker-logo.png" alt="prompt-baker" width="480">
</p> -->
![prompt-baker](assets/prompt-baker-logo.png)

# prompt-baker

Genetic optimization for **prompt + model** combinations on CSV benchmarks. You inject chat completion functions (any API, local model, LangChain agent, or heuristic), supply pools of system and user prompt templates, and search for high-scoring candidates using classification or generation metrics.

## Features

- Pluggable completion backends via `ChatModelSpec`
- Pools of system prompts, user prompts (with `{input}`), and multiple models in one run
- Tasks: **classification** and **generation**
- Metrics: accuracy, F1, precision, recall; ROUGE; optional embedding similarity and LLM-as-judge
- JSONL logging per run and helpers to plot progress and export score tables

## Requirements

- Python 3.10 or newer

## Installation

### From PyPI (pip)

```bash
pip install prompt-baker
```

### From PyPI (uv)

```bash
uv add prompt-baker
```

### From source (GitHub clone)

```bash
git clone https://github.com/YOUR_USERNAME/prompt-baker.git
cd prompt-baker
```

Then either:

**uv (recommended for this repo)**

```bash
uv sync
uv run prompt-baker --about
```

**pip in a virtual environment**

```bash
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .
prompt-baker --about
```

Replace `YOUR_USERNAME` in the clone URL with your GitHub user or organization. Before publishing, update `authors` and `project.urls` in `pyproject.toml` to match your PyPI and GitHub accounts.

## Repository layout

```
prompt-baker/
├── pyproject.toml          # Package metadata, build (hatchling), uv dev deps
├── LICENSE
├── README.md
├── src/
│   └── prompt_baker/
│       ├── __init__.py     # PromptBakerOptimizer, ChatModelSpec, OptimizerConfig
│       ├── optimizer.py
│       ├── types.py
│       ├── metrics.py
│       ├── logging.py
│       ├── visualizer.py
│       └── cli.py          # prompt-baker console entry point
├── scripts/
│   └── visualize_logs.py   # CLI to plot a run and emit scores CSV
├── tests/
└── examples/               # See examples/README.md
    ├── sentiment/
    └── rag_cat_dog/
```

## Command-line tools

After installation, the package exposes:

```bash
prompt-baker --about
```

Optimization is driven from Python. To visualize an existing run directory (contains `scores.jsonl`):

```bash
python scripts/visualize_logs.py --run-dir logs/run_YYYYMMDD_HHMMSS
```

If you installed with `uv` from the repo root, you can use `uv run python scripts/visualize_logs.py ...`.

## Python API (minimal example)

```python
from prompt_baker import ChatModelSpec, OptimizerConfig, PromptBakerOptimizer


def my_completion(system_prompt: str, user_prompt: str) -> str:
    # Inject any backend: HTTP API, LangChain agent, local model, etc.
    return "positive"


models = [
    ChatModelSpec(
        name="my-backend",
        completion_fn=my_completion,
    )
]

config = OptimizerConfig(
    task_type="classification",
    metric="accuracy",
    generations=4,
    population_size=10,
    token_length_optimisation=True,
)

optimizer = PromptBakerOptimizer(
    model_specs=models,
    system_prompts=[
        "You are a strict classifier. Return only one label.",
    ],
    user_prompts=[
        "Classify this text into {input}",
        "Given input: {input}\nReturn only class label.",
    ],
    config=config,
    paraphrase_fn=lambda prompt, concise: f"Briefly: {prompt}" if concise else f"Rewrite: {prompt}",
)

best = optimizer.optimize("data/golden.csv")
print(best)
```

### LangChain-style agent adapter

```python
def build_completion_from_agent(agent):
    def completion(system_prompt: str, user_prompt: str) -> str:
        result = agent.invoke(
            {
                "messages": [
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": user_prompt},
                ]
            }
        )
        return str(result["messages"][-1].content).strip()

    return completion
```

### Dataset contract

CSV columns (defaults can be overridden on `OptimizerConfig`):

- **input** — text fed into `{input}` in user prompts (or `input_column`)
- **target** — gold label or reference string (or `target_column`)

## Metrics

**Classification:** `accuracy`, `f1_score`, `precision`, `recall`

**Generation:** `rouge-1`, `rouge-2`, `rouge-l`; optional `embedding_similarity` (needs `sentence-transformers`); `llm_as_judge` (requires a `judge_score_fn` on the optimizer)

## Logs and visualization

Each optimization run writes a directory such as `logs/run_YYYYMMDD_HHMMSS` with:

- `events.jsonl`
- `scores.jsonl`
- `summary.json`

From the repo, generate a plot and CSV in a separate process:

```bash
uv run python scripts/visualize_logs.py --run-dir logs/run_YYYYMMDD_HHMMSS
```

Or call `plot_progress` / `create_scores_csv` from `prompt_baker.visualizer` in code (see the examples).

## Examples

Two worked examples live under `examples/`:

| Example | Directory | Summary |
|--------|-----------|---------|
| Sentiment classification | `examples/sentiment/` | CSV benchmark, genetic search over prompts and backends, optional Groq agent |
| Cat–dog RAG | `examples/rag_cat_dog/` | Chroma retrieval, multiple retriever strategies, LLM-as-judge |

See **[examples/README.md](examples/README.md)** for paths, extra dependencies, and how to run each script or notebook.

## Development

```bash
uv sync
uv run pytest
uv run ruff check src tests
```

## Contact

For questions, bug reports, or feature ideas, email [sankhoroy@gmail.com](mailto:sankhoroy@gmail.com).

## License

MIT — see [LICENSE](LICENSE).
