# {project_name} — Forge SLM Project

## READ THIS FIRST

This is a self-improving SLM training project managed by the `forge` CLI.
The flywheel runs autonomously: it trains, evaluates, patches config, generates data, and commits — all without human intervention.

If you are an AI agent starting work on this project, read this entire file before doing anything else.

---

## Your Role (if you are an AI agent)

You may be asked to:

- Run `forge flywheel --iters N` to start or resume the autonomous training loop
- Inspect `output/experiments.jsonl` to understand what has been tried
- Read `AGENT_CONTEXT.md` for the full iteration history (scores, failures, hypotheses)
- Edit `BACKGROUND.md` to add domain knowledge that improves data generation
- Edit `forge.yaml` to change hyperparameters (the flywheel will also do this automatically)
- Run `forge generate --n N` to generate more synthetic training data
- Run `forge audit --fix` to detect and fix bad training examples
- Run `forge eval` to evaluate the current adapter against the gold set
- Run `forge status` to see current accuracy and dataset sizes

**NEVER do these things:**
- Edit `dataset/gold.jsonl` — this is the locked evaluation set. Changing it invalidates all historical accuracy comparisons.
- Edit `output/best_adapter*/` — these are immutable snapshots
- Edit `AGENT_CONTEXT.md` manually — it is auto-managed by forge
- Edit `PLAN.md` manually — it is auto-managed by forge

---

## Project Files

| File | Purpose | Edit? |
|---|---|---|
| `forge.yaml` | All config — model, LoRA, training, eval, flywheel | ✅ Yes |
| `BACKGROUND.md` | Domain knowledge for Claude datagen | ✅ Yes — add insights |
| `system_prompt.md` | Exact prompt injected at train + eval time | ✅ Carefully |
| `mission.md` | Task description for datagen | ✅ Yes |
| `AGENT_CONTEXT.md` | Auto-updated iteration log | ❌ Read only |
| `PLAN.md` | Claude's strategy tracker | ❌ Auto-updated |
| `dataset/seed.jsonl` | Hand-crafted seed examples | ✅ Add examples |
| `dataset/canonical.jsonl` | Training data (grows each iteration) | ✅ forge generates/audits |
| `dataset/gold.jsonl` | **LOCKED** eval set | ❌ **NEVER TOUCH** |
| `output/experiments.jsonl` | Full experiment history | ❌ Read only |
| `output/flywheel_heartbeat.json` | Current run status | ❌ Read only |
| `output/best_score.json` | Best accuracy achieved | ❌ Read only |
| `output/best_adapter_*/` | Snapshots of best adapters | ❌ Read only |
| `llm.txt` | This file — briefing for agents | ❌ Read only |

---

## How the Flywheel Works (step by step)

1. **Claude reads all context:**
   - `BACKGROUND.md` — full domain knowledge (never truncated)
   - `AGENT_CONTEXT.md` — last ~5000 chars of iteration history
   - `PLAN.md` — current strategy and hypotheses queue
   - Live failures from the latest eval (exact inputs + expected vs actual)

2. **Claude returns an `ExperimentPlan`:**
   - `hypothesis` — plain English: what we think will help this iteration
   - `config_patches` — temporary forge.yaml overrides for THIS iteration only (not saved to disk)
   - `forge_yaml_patches` — permanent forge.yaml changes (written to disk immediately)
   - `background_additions` — new domain knowledge to append to `BACKGROUND.md`
   - `augment_focus` — what kind of examples to generate
   - `augment_n` — how many examples to generate

3. **Forge applies config patches and trains** the LoRA adapter on `dataset/canonical.jsonl`

4. **Forge evaluates** the adapter against `dataset/gold.jsonl` — this is the ground truth

5. **If accuracy improved:** snapshot adapter to `output/best_adapter_<acc>/` and update `output/best_score.json`

6. **Update context:** append to `AGENT_CONTEXT.md`, rewrite `PLAN.md` with new strategy

7. **Generate targeted training data** from eval failures using Claude + `augment_focus`

8. **Git commit** everything: data, config, context files, adapter snapshot

9. **Repeat** until `target_accuracy` reached or `max_iterations` exhausted

---

## Key Commands

```bash
forge status                    # what's the current state?
forge flywheel --iters 10       # run the full autonomous loop
forge flywheel --skip-train     # skip training first iteration (eval existing adapter)
forge generate --n 500          # generate 500 synthetic examples with Sonnet
forge generate --n 200 --batch-size 25
forge audit                     # audit dataset for errors (no changes)
forge audit --fix               # audit and fix bad examples automatically
forge eval                      # evaluate current adapter against gold set
forge train                     # train one iteration manually
forge train --epochs 3          # override epoch count
forge promote                   # snapshot current adapter manually
forge push                      # upload best adapter to HuggingFace
forge clean                     # reset augmented data (keep seed examples)
forge patch "description"       # one-shot targeted datagen
forge augment                   # augment from latest eval failures
```

---

## Crash Recovery

If the flywheel exits unexpectedly:

1. Check `output/flywheel_heartbeat.json` — `status` will be `"error"` with an `error` field
2. Run `forge status` to see last known accuracy and iteration
3. Check `output/flywheel.log` for the full traceback
4. Fix the issue (OOM → reduce batch_size; bad data → run `forge audit --fix`)
5. Resume:

```bash
# Option A: evaluate existing adapter first, then continue
forge flywheel --iters N --skip-train

# Option B: retrain from scratch at current iteration
forge flywheel --iters N
```

The flywheel is designed to resume cleanly. `AGENT_CONTEXT.md` and `PLAN.md` preserve full history across restarts.

---

## Writing Good BACKGROUND.md

This is the most important file for data quality. Claude reads it **in full** before generating every training example. The better this file, the more realistic and useful your training data.

**Required sections:**

### Task
What exactly should the model do? Be specific. Include the exact input format and the exact output format.

### Output Format
Show the **complete JSON schema** with a concrete correct example AND a concrete wrong example. If the model output is JSON, show every field, its type, and its valid values.

```
Correct: {"cmd": "land", "args": {}, "confidence": 0.99}
Wrong (missing args): {"cmd": "land", "confidence": 0.99}
Wrong (plain text): The drone should land.
```

### Edge Cases
List every tricky input you know about with the correct output. Be explicit — Claude will generate examples targeting these cases.

```
- "Return home" → cmd:rtl (NOT goto_waypoint)
- "Hover" vs "loiter" → different commands (see commands section)
- Empty input → cmd:unknown, confidence:0.0
```

### Common Failure Patterns
What has the model historically gotten wrong? Update this section as you discover patterns from eval failures. The flywheel's planner will also append to this section automatically.

```
- Drops `args` key when no args needed — must always include args: {}
- Outputs plain text when input contains a question mark
- Confuses hover and loiter
```

### What Good Examples Look Like
Describe the ideal training example: variety of input phrasing, difficulty range, output correctness criteria.

```
Good inputs: vary formality, length, and ambiguity.
Good outputs: always valid JSON, cmd from exact allowed list, args always present.
Difficulty: 30% easy, 50% medium, 20% hard.
```

---

## forge.yaml Quick Reference

```yaml
model:
  base: unsloth/functiongemma-270m-it  # HuggingFace model ID
  max_seq_len: 2048
  load_in_4bit: true

lora:
  r: 32           # rank — higher = more capacity, slower
  alpha: 32       # usually = r
  dropout: 0.05

training:
  epochs_min: 2
  epochs_max: 4
  batch_size: 4
  learning_rate: 5.0e-5
  gradient_accumulation_steps: 1
  warmup_ratio: 0.1
  weight_decay: 0.01
  lr_scheduler: cosine
  seed: 42

eval:
  target_accuracy: 0.95   # stop when this is reached
  scorer: json_cmd        # exact | json_cmd | custom (scorer.py)
  max_new_tokens: 256

dataset:
  seed: dataset/seed.jsonl
  gold: dataset/gold.jsonl        # LOCKED — never train on this
  canonical: dataset/canonical.jsonl

flywheel:
  max_iterations: 10
  augment_per_failure: 20
  datagen_model: claude-sonnet-4-6
  planner_model: claude-sonnet-4-6

compute:
  device: cuda:0
```

---

## Understanding Accuracy Numbers

- `forge eval` reports accuracy as a fraction of gold examples the model answered correctly
- The scorer type (`eval.scorer`) defines "correct":
  - `exact` — exact string match
  - `json_cmd` — parses JSON, checks only the `cmd` field
  - `custom` — calls `scorer.py` in the project root
- Always compare accuracy against the **same gold set** — never modify `gold.jsonl` after training starts
- Accuracy in `output/experiments.jsonl` is comparable across all iterations

---

## Understanding the Git History

Every flywheel iteration is committed. Run `git log --oneline` to see the full training history:

```
a1b2c3d forge iter 8: 93.2% — Increasing LoRA rank to 64 for capacity
b2c3d4e forge iter 7: 91.5% — Targeting hover/loiter confusion (60 examples)
c3d4e5f forge iter 6: 88.1% — Reducing LR after loss oscillation
...
```

To see what changed in a specific iteration:
```bash
git show a1b2c3d --stat    # files changed
git show a1b2c3d           # full diff
```

---

## Monitoring a Running Flywheel

```bash
# Live status
watch -n 10 cat output/flywheel_heartbeat.json

# Accuracy progression
cat output/experiments.jsonl | python3 -c "
import sys, json
for line in sys.stdin:
    r = json.loads(line)
    print(f\"iter {r['iteration']}: {r.get('accuracy', 0):.1%} — {r.get('hypothesis','')[:60]}\")
"

# Latest failures
cat output/eval_failures.jsonl | python3 -m json.tool | head -60

# Agent's current strategy
cat PLAN.md

# Full iteration history
tail -200 AGENT_CONTEXT.md
```
