Metadata-Version: 2.4
Name: raftaar
Version: 0.1.0
Summary: Know what your robot demonstrations will teach, before you train. Predicts which policy class will fail on a dataset, and where.
Author: Aslan Satary Dizaji
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/aslansd/raftaar
Project-URL: Repository, https://github.com/aslansd/raftaar
Project-URL: Issues, https://github.com/aslansd/raftaar/issues
Keywords: robotics,behaviour-cloning,imitation-learning,dataset-quality,demonstrations,lerobot,data-curation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: scipy>=1.11
Requires-Dist: scikit-learn>=1.4
Provides-Extra: plot
Requires-Dist: matplotlib>=3.8; extra == "plot"
Provides-Extra: validate
Requires-Dist: matplotlib>=3.8; extra == "validate"
Provides-Extra: web
Requires-Dist: flask>=3.0; extra == "web"
Requires-Dist: gunicorn>=22.0; extra == "web"
Provides-Extra: provenance
Requires-Dist: daftar>=0.8; extra == "provenance"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: matplotlib>=3.8; extra == "dev"
Provides-Extra: all
Requires-Dist: matplotlib>=3.8; extra == "all"
Requires-Dist: flask>=3.0; extra == "all"
Requires-Dist: gunicorn>=22.0; extra == "all"
Requires-Dist: daftar>=0.8; extra == "all"
Dynamic: license-file

# Raftaar

**رفتار** — *behaviour.* The same word in Persian, Turkish and Urdu.

**Know what your demonstrations will teach, before you train.**

Raftaar reads a robot demonstration dataset and predicts *which policy class
will fail on it, and where* — without training anything. It runs on a CPU in
minutes.

Structural validators ("is this file corrupt, are the camera intrinsics valid")
are commoditising fast. Raftaar works one level up, on the **distribution**:
the geometry of the demonstrations themselves, and what a learning algorithm
will converge to when trained on them.

```bash
pip install raftaar
```

Python 3.10+. Apache 2.0. Three dependencies — numpy, scipy, scikit-learn. No
GPU, no torch, no downloads.

---

## The failure that matters

A dataset containing two valid but incompatible strategies — half the operators
reach around an obstacle on the left, half on the right — will train a
regression policy to do neither. The policy converges to the average, the
average is a path nobody demonstrated, and **the training loss goes down the
entire time**.

Validation loss does not catch it either, because the validation split has the
same structure as the training split. The same is true of coverage holes, of
silent recalibrations mid-collection, and of actuator channels that stopped
moving. Every one of these is detectable before training. None of them is
currently detected.

### Where this sits

The robot-data tooling layer is filling in from the bottom. Structural linters
that check schemas, timestamps and decodability already exist and are improving
fast — `trajlens` is one, and it audits the public LeRobot Hub today. That layer
is commoditising and Raftaar does not compete with it.

Raftaar works on the layer above: not "is this file valid" but **"what will a
policy learn from this distribution, and which policy class will fail"**. The
two are complementary — run a structural linter first, then this.

---

## Quickstart

```bash
# 1. A dataset with known, deliberately injected faults
raftaar synth field_data --episodes 120 \
    --faults bimodal_detour coverage_hole demonstrator_jitter camera_shard dead_actuator

# 2. Diagnose it — no training
raftaar scan datasets/field_data --out reports/field_data

# 3. Check the diagnosis was right — train policies and roll them out
raftaar validate datasets/field_data --trials 40
```

From Python:

```python
from raftaar import load_dataset, scan

report = scan(load_dataset("datasets/field_data"))
for finding in report["findings"]:
    print(finding["severity"], finding["id"], finding["where"])
    print("   ", finding["what"])
```

---

## The five detectors

| Detector | Question it answers |
|---|---|
| `AVERAGING_HAZARD` | Do demonstrations contain competing strategies whose *average* is a trajectory nobody performed? |
| `COVERAGE_GAP` | Which task conditions were never demonstrated? (sample-size aware — sparsity is not a hole) |
| `ACTION_INCONSISTENCY` | Do different episodes, in near-identical states, disagree on what to do? |
| `REPRESENTATION_SHARD` | Was part of this dataset recorded under a different calibration or lighting? |
| `IDLE_CHANNEL` | Which action channels carry no signal? (sharpened with `--reference`) |

`AVERAGING_HAZARD` is the differentiated one. It clusters demonstrations into
strategies, measures how far their average sits from the nearest real strategy
in units of within-strategy spread, and **suppresses splits that the task
conditions themselves explain** — paths to objects on the left and on the right
*should* differ, and that is the policy's job, not a defect.

---

## Why the naive version doesn't work

Three things had to be right before any of this detected anything:

1. **Strategy signatures.** Comparing raw trajectories fails: variation in
   *where the object was* swamps variation in *how the operator got there*.
   Subtracting the straight line between a phase's own endpoints isolates
   strategy from condition.
2. **A confound guard.** A mode split that the initial conditions predict is
   suppressed, or every well-collected dataset would be flagged.
3. **Elbow selection, not BIC minimum.** BIC falls monotonically as components
   are added, because real demonstration clusters always have sub-structure.

---

## Does it actually predict anything?

`raftaar validate` trains a unimodal regression policy (stands in for
ACT-style behaviour cloning) and a mode-conditioned policy (stands in for
diffusion/flow policies) on the same data and rolls both out. 120 episodes,
40 trials.

| Dataset | Raftaar verdict | Unimodal BC | Mode-conditioned BC |
|---|---|---|---|
| `clean` | no warnings | **100% success, 0% collision** | 78% success, 15% collision |
| `bimodal_only` | AVERAGING_HAZARD | 75% success, **25% collision** | **100% success, 0% collision** |
| `field_data` | 5 findings | 20% success, **80% collision** | 88% success, 5% collision |

The prediction runs both ways. On clean data the tool stays quiet and the simple
policy wins outright — reaching for a multimodal policy there makes things
*worse*. On bimodal data the tool warns, and the simple policy drives into the
obstacle. That symmetry is what makes the output actionable rather than
decorative.

### A caveat on small datasets

"No warnings" on clean data holds at 120 episodes. Below roughly 90,
`ACTION_INCONSISTENCY` fires at *warning* severity on clean data, because
near-identical states genuinely do disagree when there are few episodes per
region of the state space. The detector is measuring something real; it is
under-powered, not wrong. Both behaviours are pinned by tests in
`tests/test_detectors.py` so the boundary is documented rather than folklore.

---

## Optional extras

The core install is deliberately small. Everything else is opt-in:

```bash
pip install "raftaar[plot]"        # matplotlib, for the diagnostic figure
pip install "raftaar[validate]"    # the rollout study
pip install "raftaar[provenance]"  # record what produced each scan
pip install "raftaar[web]"         # the browser demo in web/
pip install "raftaar[all]"
```

`raftaar scan` writes its markdown report and JSON with no plotting stack
installed, and tells you what it skipped. A dataset audit should not require
matplotlib to run.

### Provenance

A scan is a measurement and `validate` is an experiment, so both depend on
seeds, library versions and the exact dataset that went in. With
[daftar](https://pypi.org/project/daftar/) installed:

```python
from raftaar.provenance import tracked_scan

report = tracked_scan("datasets/field_data", label="field-data-audit")
```

This records the dataset's content hash, the Raftaar version, every detector's
headline number and all findings, so two scans of the same dataset can be diffed
rather than eyeballed. Without daftar it behaves exactly like
`scan(load_dataset(path))` — the integration is a no-op, not an error.

---

## Tests

```bash
pip install -e ".[dev]"
pytest -q          # 29 tests
```

The detector tests are the ones that matter. Every fault is *injected
deliberately* into the synthetic environment, so each detector is scored against
ground truth rather than opinion — and both directions are tested:

- **sensitivity** — a dataset built with a fault must raise that fault's finding
- **specificity** — clean data must stay quiet, because a detector that fires on
  everything is worse than no detector

---

## Status

Research prototype, honestly labelled. The synthetic environment exists so the
detectors can be scored against ground truth. The next milestone is a
`LeRobotDataset` adapter and a run across the public dataset hub — at which
point the claims above get tested against data nobody generated to order.

## Licence

Apache 2.0.
