Metadata-Version: 2.4
Name: harbor-rewardkit
Version: 0.2.1
Summary: Lightweight grading toolkit for environment-based tasks.
Keywords: grading,evaluation,rewards,llm,agents,benchmarks
Author: benediktstroebl
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Dist: litellm>=1.83
Requires-Dist: pillow>=10.0 ; extra == 'all'
Requires-Dist: markitdown[pdf,docx,pptx,xlsx,xls,outlook]>=0.1 ; extra == 'all'
Requires-Dist: markitdown[pdf,docx,pptx,xlsx,xls,outlook]>=0.1 ; extra == 'documents'
Requires-Dist: pillow>=10.0 ; extra == 'image'
Requires-Python: >=3.12
Project-URL: Repository, https://github.com/harbor-framework/harbor
Project-URL: Issues, https://github.com/harbor-framework/harbor/issues
Provides-Extra: all
Provides-Extra: documents
Provides-Extra: image
Description-Content-Type: text/markdown

# Harbor Rewardkit

[![](https://dcbadge.limes.pink/api/server/https://discord.gg/QVvyhRw5UQ)](https://discord.gg/QVvyhRw5UQ)
[![Docs](https://img.shields.io/badge/Docs-000000?style=for-the-badge&logo=mdbook&color=105864)](https://harborframework.com/docs/rewardkit)

Rewardkit is a lightweight verifier package for Harbor tasks or standalone use.

## Installation

```bash
uv tool install harbor-rewardkit
```

## Programmatic verifier

```python
# tests/checks.py
from rewardkit import criteria

criteria.file_exists("output.txt")
criteria.file_contains("output.txt", "hello")
```

Run it with:

```bash
uvx --from harbor-rewardkit rewardkit /tests
```

Rewardkit runs the criteria against `/app` and writes
`/logs/verifier/reward.json`. With files directly under `tests/`, their scores
are combined into one `reward` automatically.

## Configure scoring

Each Python file that registers criteria is one equal-weighted component named
after its filename stem. Use `[scoring.<stem>]` in `reward.toml` to change how
the criteria within that file combine:

```toml
# tests/reward.toml
[scoring.checks]
aggregation = "all-pass"
```

Files that only provide imports or `shared=True` criterion definitions and
register no checks are ignored.

## Judge verifier

```toml
# tests/quality.toml
[judge]
judge = "anthropic/claude-sonnet-5"
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"
```

Python files and judge TOMLs can coexist. A judge TOML is also a component named
after its stem and has its own optional `[scoring]` section.

To change how these flat components combine, add a named reward aggregation:

```toml
# tests/reward.toml
[[reward]]
name = "reward"
aggregation = "weighted-mean"
weights = { checks = 2.0, quality = 1.0 }
```

## Multi-reward verifier

Put scoring inputs in subdirectories to emit separate dimensions:

```text
tests/
├── correctness/
│   └── checks.py
├── structure/
│   └── files.py
└── quality/
    └── judge.toml
```

This emits `correctness`, `structure`, and `quality`. Add a root
`tests/reward.toml` to combine them into a main reward:

```toml
[[reward]]
name = "reward"
aggregation = "weighted-mean"
weights = { correctness = 2.0, structure = 1.0, quality = 1.0 }
```

Inside a dimension, one unnamed `[[reward]]` can set weights across local
Python files, judge TOMLs, and child directories. Reference files by stem and
child groups by directory name.

See the [documentation](https://harborframework.com/docs/rewardkit) for custom
criteria, agent judges, MCP servers, nested groups, and the complete config
reference. A [working Harbor task](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/reward-kit-example)
shows the full layout.
