Metadata-Version: 2.4
Name: datahive-tools
Version: 0.1.4
Summary: Client tooling to collect, validate, annotate and upload HiveBoard manipulation episodes.
Author: HiveBoard
License: MIT
Project-URL: Homepage, https://github.com/hiveboard-bench/DataHive
Project-URL: Issues, https://github.com/hiveboard-bench/DataHive/issues
Keywords: robotics,manipulation,dataset,hiveboard,hdf5
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: huggingface_hub>=0.24
Requires-Dist: h5py>=3.10
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.24
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn>=0.29
Requires-Dist: typer>=0.12
Requires-Dist: python-multipart>=0.0.9
Provides-Extra: video
Requires-Dist: opencv-python-headless>=4.8; extra == "video"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: opencv-python-headless>=4.8; extra == "dev"
Dynamic: license-file

# DataHive

`datahive-tools`: the client-side package a lab installs to collect,
validate, annotate, and upload HiveBoard manipulation episodes to its
private Hugging Face dataset repo.

> **Credits & Attribution**: All pipeline architecture, workflows, and core design in DataHive are directly adapted from and credited to [**Oopsie Data**](https://github.com/oopsie-data) (`oopsie-data` / `oopsie_data_tools`). Everything in DataHive — from the CLI workflow, robot profiles, and dataset indexing to the annotation GUI and Hugging Face Hub synchronization pipeline — was built upon the pipeline and concepts developed by the Oopsie Data team.

## Install

```
pip install datahive-tools
pip install "datahive-tools[video]"   # optional: OpenCV, so videos are actually checked
```

Requires Python 3.10 or newer. This installs the `datahive` command.

## Quickstart

```
datahive init                 # one-time: writes ~/.datahive/config.yaml (0600)
datahive new-profile          # writes samples/robot_profile.yaml -- fill in ONCE
# ... record episodes into samples/{session}/episodes/{episode}.h5 + .mp4 ...
datahive check <episode_id>   # pre-annotation health check (frames, Hz, MP4s)
datahive annotate <episode_id>
datahive validate <episode_id>
datahive upload <episode_id>
datahive sync                 # reconcile all of samples/ with the Hub
datahive list
datahive delete <episode_id>
datahive interface --port 8000 # local web GUI, localhost only (alias: serve)
```

## The sections of `datahive interface`

Opening the interface shows a start screen with two sections, plus an About and FAQ area. Both sections share the same `samples/` tree.

- **Runner** – runs HiveBoard trials. Create a session (operator and date; the lab ID comes from `datahive init` and the platform ID from the robot profile), generate the trial plan (5 trials for each chosen task, shuffled or ordered), then run each trial in one of two modes, switchable at any time:
  - **Manual**: 5 s countdown, stopwatch, sounds and automatic timeout in the browser (Space starts and stops), then record the outcome and **upload the HDF5 file and one video per camera in the robot profile**. The trial counts as valid only when all of them are uploaded and the episode passes validation.
  - **Automatic**: send the trial to your robot script (below). It records the episode, and you annotate it in the Annotate section, which opens on that episode.
- **Annotate** – review recorded episodes, annotate them, validate the data, and upload it to the Hub.

Robot side of automatic mode:

```python
from datahive.collect import CollectClient

robot = CollectClient("samples", base_url="http://127.0.0.1:8000")
while True:
    task = robot.wait_for_task()             # blocks until you press "Send to robot"
    with robot.new_writer(task) as writer:   # EpisodeWriter for that trial
        ...                                  # append_proprioception / append_command / attach_video
    robot.finish(writer)                     # hands the episode over, waits for the annotation
```

Trials are saved as rows of `samples/<session>/trials.csv`; all times are stored in UTC.

## AI assistant skills

Two skills for AI coding assistants (e.g. Claude Code) live in `src/datahive/skills/`:

- `datahive-data-prep` – how to lay out, format, annotate and check data under `samples/` so it validates.
- `datahive-auto-collect` – how automatic collection works and how to adapt the robot script to your setup.

Install them for Claude Code with:

```
datahive install-skill          # into ./.claude/skills (this project)
datahive install-skill --user   # into ~/.claude/skills (all projects)
datahive install-skill --dir DIR --force
```

## Development

```
git clone https://github.com/hiveboard-bench/DataHive
cd DataHive
pip install -e ".[dev]"
pytest
```

All tests run with no real network access and no real Hugging Face
credentials.

## Credits & Acknowledgments

All credit for the underlying pipeline architecture goes to the **Oopsie Data** (`oopsie-data` / `oopsie_data_tools`) project. DataHive was built on top of their pipeline patterns, including:

- **Interactive Annotation & GUI**: The web-based annotator interface, video playback controls, task picker, episode navigation, and failure classification.
- **Data Lifecycle**: The full `init` -> `new-profile` -> `annotate` -> `validate` -> `upload` -> `sync` pipeline.
- **Dataset Structure & Profiling**: Hardware platform specification snapshots, HDF5 episode storage, and Hub reconciliation.

We are deeply grateful to the Oopsie Data contributors for their pioneering work in robotic data pipelines.
