Metadata-Version: 2.4
Name: smithtune
Version: 0.0.1
Summary: Utilities for preparing conversational datasets.
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# smithtune

Utilities for preparing conversational datasets. Runs locally with no runtime dependencies.

## Install

Requires Python 3.10 or newer.

```bash
pip install smithtune
```

## Split a dataset

```bash
smithtune split data.jsonl --group-by conversation_id --output-dir splits/
```

Each input line must be a JSON object with a top-level group field containing a nonempty string or integer:

```json
{"conversation_id":"a","messages":[{"role":"user","content":"Hello"}]}
{"conversation_id":"a","messages":[{"role":"assistant","content":"Hi"}]}
{"conversation_id":"b","messages":[{"role":"user","content":"Good morning"}]}
```

Creates `train.jsonl`, `validation.jsonl`, and `test.jsonl` and prints record counts. Records with the same group ID always go into the same file. Use IDs that are unique across your input sources.

Defaults target 80% training, 10% validation, and 10% test **by group**. Hash-based assignment gives approximate proportions; small datasets may have empty splits. Adding or reordering records does not change a group's assignment when its ID, seed, and fractions stay the same.

```bash
smithtune split data.jsonl --group-by conversation_id --output-dir splits-v2/ \
  --validation-fraction 0.15 --test-fraction 0.15 --seed 42
```

The splitter preserves record bytes and order within each output file, adding a final newline if needed. It validates the entire input in memory before writing. Blank lines, invalid JSON, duplicate JSON keys, and missing or invalid group IDs are rejected with a line number. String IDs and integer IDs are distinct.

The output directory must not exist, and its parent directory must exist. Existing files are never overwritten. No data is uploaded or fetched.

## Python

```python
from smithtune import split_jsonl

counts = split_jsonl("data.jsonl", "splits", group_by="conversation_id")
```

## Development

```bash
PYTHONPATH=src python -m unittest discover -s tests -v
```
