Metadata-Version: 2.3
Name: syvain-training-utils
Version: 0.0.246
Summary: Shared runtime, diagnostics, and checkpoint utilities for Syvain training runs
Requires-Dist: numpy>=2.5.1
Requires-Dist: obstore>=0.11.0,<0.12.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: torch>=2.13.0
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: ruff>=0.15.12 ; extra == 'dev'
Requires-Dist: ty>=0.0.34 ; extra == 'dev'
Requires-Python: >=3.14, <3.15
Provides-Extra: dev
Description-Content-Type: text/markdown

# syvain-training-utils

Internal [Syvain](https://syvain.com/) helpers for small, explicit ML training
runs. No secret sauce here, just shared runtime, device-diagnostic, and
checkpoint patterns.

## Install

```bash
uv add syvain-training-utils
```

## Runtime setup

```python
from syvain_training_utils import (
    generate_run_id,
    require_torch_compile_toolchain,
    select_device,
)

run_id = generate_run_id()
device = select_device()
require_torch_compile_toolchain()
```

## Load a project-042 initialization bundle

Create the uncompiled BF16 model first, then verify and load the immutable
model-only state before constructing the optimizer:

```python
import torch

from syvain_training_utils import load_initialization_bundle_variant

# `storage_config`, the manifest pins, and `model_config` come from the sealed
# project-042 run contract. Construct `model` in BF16 without compiling it.
initialization = load_initialization_bundle_variant(
    storage_config=storage_config,
    manifest_key=INITIALIZATION_MANIFEST_KEY,
    expected_manifest_size=INITIALIZATION_MANIFEST_SIZE,
    expected_manifest_sha256=INITIALIZATION_MANIFEST_SHA256,
    variant_id=INITIALIZATION_VARIANT_ID,
    expected_config=model_config,
    model=model,
)
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
```

The loader verifies the manifest, frozen recipe and source pins, variant ledger,
model object, and destination model. It accepts only a bare BF16 state dict,
loads it strictly, and returns an immutable receipt for the Metrics config.

## Device smoke test

```python
import json

from syvain_training_utils import run_device_smoke_test

report = run_device_smoke_test(require_cuda=True)
print(json.dumps({"smoke": report}, indent=2, sort_keys=True))
```

## Checkpoint a training run

```python
from syvain_training_utils import (
    StorageConfig,
    TrainingLoopState,
    load_training_checkpoint_if_available,
    save_model_checkpoint,
)

storage_config = StorageConfig(
    bucket="my-training-bucket",
    s3_base_url="https://t3.storage.dev",
    region="auto",
    access_key_id="...",
    secret_access_key="...",
)
checkpoint_base_path = "models/my-model"

loop_state = TrainingLoopState(
    global_step=global_step,
    curriculum_stage=curriculum_stage,
    curriculum_step=curriculum_step,
)

save_model_checkpoint(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    experiment_slug=experiment_slug,
    run_id=run_id,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    loop_state=loop_state,
    checkpoint_label=f"step-{global_step:012d}",
)

resume = load_training_checkpoint_if_available(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    device=device,
)
```

The library owns the object-store clients. Every transient retry opens a fresh
client, and expired Tigris multipart sessions restart the complete upload at the
same checkpoint key. Checkpoint bodies are written through a temporary local
file, uploaded with an adaptive multipart size, and downloaded with resumable
range reads. Ensure the machine has temporary disk capacity for one checkpoint.

The manifest remains a pointer to the current checkpoint key within the
configured bucket and also records its byte size, ETag, and SHA-256 digest. The
manifest is published only after the checkpoint upload succeeds. Loading
verifies the complete digest before deserializing the model, optimizer, optional
scheduler, and PyTorch RNG state.

The storage transport uses a 10-second connect timeout, a 60-second
read-inactivity timeout, and a 10-minute overall request timeout. A bounded
15-minute outer no-progress retry window owns recovery and client replacement.

## Load an immutable initialization bundle

Create the BF16 model before its optimizer. Then load one variant using the
published manifest pin and the exact configuration recorded by the bundle.

```python
from syvain_training_utils import load_initialization_bundle_variant

receipt = load_initialization_bundle_variant(
    storage_config=storage_config,
    manifest_key=INITIALIZATION_MANIFEST_KEY,
    expected_manifest_size=INITIALIZATION_MANIFEST_SIZE,
    expected_manifest_sha256=INITIALIZATION_MANIFEST_SHA256,
    variant_id=INITIALIZATION_VARIANT_ID,
    expected_config=INITIALIZATION_CONFIG,
    model=model,
)
```

The loader verifies canonical manifest and ledger bytes, complete object and
tensor digests, BF16 shapes, scalar counts, strict model keys, and untied
embedding and output-head storage. It returns an immutable receipt. A run does
not need a teacher-model or Safetensors dependency.
