Metadata-Version: 2.4
Name: worldproof
Version: 0.1.0
Summary: A reality check for world models: diagnose where and why rollout predictions break.
Project-URL: Homepage, https://github.com/BuceaGeorgia/worldproof
Project-URL: Repository, https://github.com/BuceaGeorgia/worldproof
Project-URL: Issues, https://github.com/BuceaGeorgia/worldproof/issues
Project-URL: Changelog, https://github.com/BuceaGeorgia/worldproof/blob/main/CHANGELOG.md
Author: worldproof contributors
License: Apache-2.0
License-File: LICENSE
Keywords: diagnostics,evaluation,video-prediction,world-models
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=9.5
Requires-Dist: torch>=2.1
Provides-Extra: atari
Requires-Dist: ale-py>=0.10; extra == 'atari'
Requires-Dist: gymnasium>=1.0; extra == 'atari'
Provides-Extra: fidelity
Requires-Dist: lpips>=0.1.4; extra == 'fidelity'
Provides-Extra: fvd
Requires-Dist: torchvision>=0.15; extra == 'fvd'
Provides-Extra: io
Requires-Dist: av>=12.0; extra == 'io'
Requires-Dist: imageio-ffmpeg>=0.4; extra == 'io'
Requires-Dist: imageio>=2.31; extra == 'io'
Provides-Extra: lerobot-data
Requires-Dist: huggingface-hub>=0.20; extra == 'lerobot-data'
Requires-Dist: imageio-ffmpeg>=0.4; extra == 'lerobot-data'
Requires-Dist: imageio>=2.31; extra == 'lerobot-data'
Requires-Dist: pyarrow>=14; extra == 'lerobot-data'
Provides-Extra: swm
Requires-Dist: hydra-core>=1.3; extra == 'swm'
Requires-Dist: imageio>=2.31; extra == 'swm'
Requires-Dist: opencv-python-headless>=4.9; extra == 'swm'
Requires-Dist: stable-pretraining>=0.1.8; extra == 'swm'
Requires-Dist: stable-worldmodel==0.1.1; extra == 'swm'
Requires-Dist: transformers<5; extra == 'swm'
Description-Content-Type: text/markdown

# worldproof

[![CI](https://github.com/BuceaGeorgia/worldproof/actions/workflows/ci.yml/badge.svg)](https://github.com/BuceaGeorgia/worldproof/actions/workflows/ci.yml)

**A reality check for world models.**

A world model predicts the future of a scene from a starting context and a
sequence of actions. worldproof looks at those predictions and tells you where
and why they go wrong. It compares a model's rollout against ground truth and
against physical rules (does an object vanish, does the count of objects change,
does the model react to the action at all), then writes a report card.

It does not score task success, planning quality, or how nice the video looks.
That is a different job. See [SPEC.md](SPEC.md) for the exact scope.

| Question | Who answers it |
|---|---|
| Does the acting model succeed at the task? | VLABench, LIBERO |
| Can the model be trained and used for planning? | stable-worldmodel |
| Is the prediction correct, and where does it break? | **worldproof** |

Status: alpha, v0.1 in development. The API and the on-disk rollout format are
still moving. The data contracts that are frozen are listed in
[SPEC.md](SPEC.md) section 5.

## See a report card

Here is a report card for a copy-the-last-frame baseline on Push-T, a
manipulation benchmark that runs in a simulator. Global PSNR looks high (the
scene barely moves), but the dynamic-region score slides down over the horizon:
the baseline cannot predict the one thing that actually moves, the pushed block.
That gap is the whole point of the tool. It also runs on real camera footage,
see the SO-101 example further down.

![worldproof report card on Push-T](https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/lerobot-pusht.png)

<!-- GIF idea: a short screen capture of `worldproof evaluate` running in a
terminal (the verdict printing, the report being written) would sit well here. -->

## Install

```bash
pip install worldproof
```

Until the first PyPI release lands, install straight from git:

```bash
pip install "worldproof @ git+https://github.com/BuceaGeorgia/worldproof"
```

The core install is small on purpose: numpy, torch, and pillow only, so
`worldproof evaluate` runs on a laptop with no GPU. Install the torch build that
fits your machine first (CPU, CUDA, or Apple Silicon MPS). worldproof never pins
a CUDA-specific torch. See [pytorch.org](https://pytorch.org/get-started/locally/).

Heavier features are optional extras. They are imported only when you use them,
never at import time.

| Extra | What it adds |
|---|---|
| `worldproof[fidelity]` | the LPIPS perceptual metric |
| `worldproof[fvd]` | the default FVD extractor (torchvision Kinetics r3d_18, opt in with `evaluate --fvd`) |
| `worldproof[swm]` | the stable-worldmodel adapter and its gym sim oracle |
| `worldproof[atari]` | `AtariSimOracle`, a deterministic Atari sim oracle (CPU, ROMs bundled) |
| `worldproof[lerobot-data]` | `LeRobotDatasetSource`, reads LeRobotDataset v3.0 (parquet and mp4) as ground truth, no `lerobot` package, works on Python 3.10 |
| `worldproof[io]` | reading video-backed rollout folders |

## Quickstart (under a minute, no download)

This makes a handful of rollouts with the built-in toy simulator and a naive
baseline, scores them, and writes a report card. Core dependencies only. The
full script is [`examples/quickstart.py`](examples/quickstart.py), run in CI on
Ubuntu, macOS, and Windows.

```python
import json, tempfile
from pathlib import Path
import numpy as np
from worldproof import (
    ToySimOracle, CopyLastFrameBaseline, make_rollout, save_rollout,
    iter_rollouts, evaluate, report_json, report_html, Capabilities,
)

out = Path(tempfile.mkdtemp())
oracle, model = ToySimOracle(size=48), CopyLastFrameBaseline()
rng = np.random.default_rng(0)
for i in range(6):
    actions = rng.uniform(-2, 2, (6, 2)).astype(np.float32)
    save_rollout(make_rollout(oracle, model, seed=i, actions=actions, n_samples=2),
                 out / "rollouts" / f"ep_{i:02d}")

rollouts = list(iter_rollouts(out / "rollouts"))
report, run_report = evaluate(rollouts, capabilities=Capabilities.detect(has_tracker=True))
(out / "report.json").write_text(json.dumps(report_json(report), indent=2))
(out / "report.html").write_text(report_html(report, rollouts, run_report))
print(report.verdict)
```

## Two commands

worldproof has two commands, kept separate on purpose. `generate` may be heavy
(it runs a model). `evaluate` stays light and never runs a model, so it always
works on a laptop.

```bash
# run a model and a sim oracle to produce a folder of rollouts
worldproof generate --sim toy --model action-blind --n 8 --out rollouts

# score them (runs the metrics your machine supports, reports what it skipped)
worldproof evaluate rollouts --json report.json --html report.html
```

`--model` accepts `copy-last-frame`, `action-blind`, or `swm:<checkpoint>` (a
real latent world model, needs `worldproof[swm]`). `--sim` accepts `toy` or
`gym:<env-id>` (for example `gym:swm/TwoRoom-v1`).

`evaluate` runs the metrics your machine supports and reports what it skipped and
why. A MacBook gets a partial but real report, never an install error.

## Examples

Runnable examples, each writes a `report.json` and a `report.html`.

### Atari (game frames from an emulator, no download)

```bash
pip install worldproof[atari]
python examples/atari_demo.py
```

The Atari emulator is deterministic and runs on CPU, so it gives varied game
pixels that exercise the metrics on a laptop. On Pong the copy-last-frame
baseline scores badly on fidelity because the ball moves every frame. Atari
sprites sit on a plain background, so this example also turns on the built-in
tracker, which lets the object invariants (count conservation and permanence)
run alongside the fidelity metrics.

![worldproof report on Atari Pong](https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/atari-pong.png)

<!-- GIF idea: a side-by-side loop of the predicted rollout next to the true
future for one Pong episode would show the failure clearly. -->

### LeRobot dataset, simulated or real

```bash
pip install worldproof[lerobot-data]
python examples/lerobot_demo.py                                          # lerobot/pusht (a simulator)
python examples/lerobot_demo.py --repo Qiu-Xinchuan/so-101_pen-transfer  # a real SO-101 arm
python examples/lerobot_demo.py --repo <any LeRobotDataset v3.0 on the Hub>
```

This reads a LeRobotDataset v3.0 straight from its parquet and mp4 files, so it
works without the `lerobot` package and runs on Python 3.10. That opens up the
whole LeRobot Hub, both simulated benchmarks and real robot recordings.

The Push-T report at the top of this page comes from a simulator. Here is the
same pipeline on real cameras: a physical SO-101 arm doing a pen transfer, three
cameras at 480 by 640. The baseline scores high even on the moving regions,
because the arm moves slowly over this short horizon. That is a useful result by
itself: to tell models apart on slow real footage you need a longer horizon or a
more dynamic task.

![worldproof report card on the real SO-101 pen-transfer dataset](https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/so101-real.png)

<!-- GIF idea: a loop of a real SO-101 episode (context, then predicted vs true
future) would show how little the scene changes over the horizon. -->

### Real latent world model (LeWM / DINO-WM)

```bash
pip install worldproof[swm]
python examples/pusht_demo.py
```

This runs the real `quentinll/lewm-pusht` checkpoint. A latent model predicts in
its own encoded space, so the report shows the latent metrics (latent prediction
error and action recoverability) rather than pixel scores. Read the notes in the
script: driving it on real trajectories needs a dataset step that is still open,
so the example ships with a stand-in dataset that proves the pipeline against the
real model.

![worldproof report for the lewm-pusht latent model](https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/pusht-lewm-latent.png)

## What it measures

- Pixel fidelity: PSNR and SSIM (pure numpy) and LPIPS (an extra), each as a
  curve over the horizon and again on the moving regions only, so a static
  background cannot inflate the score.
- Latent prediction error for latent models, in the space the model predicts in.
- Action recoverability, the main latent check: can the actions be recovered from
  the predicted latents? A latent space can look sharp and still fail this.
- Calibration: does the spread across samples match the actual error (ECE, MCE)?
- The signature checks: counterfactual divergence (same start, two action
  sequences, does the predicted change match reality) and failure faithfulness
  (does the model reproduce a real failure or imagine success), fed by a sim
  oracle that produces true futures for any action sequence.
- Invariants: object count conservation, and object permanence through occlusion.
- FVD, reported as a weak reference (it tracks video quality, not dynamics). The
  math ships and is tested. A default feature extractor ships too (torchvision
  Kinetics r3d_18, opt in with `evaluate --fvd`), or you can pass your own for a
  paper-comparable backbone.

Every metric ships with a corruption test it responds to and passes a ranking
test (a real model beats a naive baseline beats a broken one). Scores across
rollouts use the interquartile mean with bootstrap confidence intervals, not a
bare mean and standard deviation.

## Not done yet

- FVD's default extractor is a torchvision Kinetics r3d_18, not the I3D used in
  most published FVD, so its numbers are not comparable to those. It is a weak
  reference either way. Pass your own extractor for a paper-comparable backbone.
- `LeRobotDatasetSource` reads v3.0 datasets that store frames as mp4. Older v2.x
  layouts and datasets that store frames in parquet are follow-ons. mp4 frames
  are lossy, which the report can note.
- The tracker behind the invariants is a clean-scene numpy tracker. Messy real
  video needs a stronger tracker, which is deferred.
- Driving the real latent checkpoint on real trajectories needs a dataset step
  that reproduces the model's action encoding. The reusable provider is shipped;
  that last piece is open.

## Pinned versions and support

- `worldproof[swm]` pins `transformers<5`. The published checkpoints predate the
  transformers 5.x weight rename, which breaks loading.
- The `lerobot` model adapter is not supported. Its world models cannot be driven
  by an external action sequence into our rollout, and they need a lot of GPU
  memory. The LeRobot data path is supported through `LeRobotDatasetSource`.

## Contributing

Development uses [uv](https://docs.astral.sh/uv/). Consumers never need it.

```bash
uv sync --all-extras --group dev
uv run pytest
ruff check --fix && ruff format
```

## License

Apache-2.0.
