Metadata-Version: 2.4
Name: gidata
Version: 0.3.0
Summary: GI Labs Data Platform CLI — upload and manage robotics data
Project-URL: Homepage, https://github.com/General-Intelligence-Labs/gilabs-data-center
Project-URL: Repository, https://github.com/General-Intelligence-Labs/gilabs-data-center
Project-URL: Issues, https://github.com/General-Intelligence-Labs/gilabs-data-center/issues
Author: General Intelligence Labs
License: Proprietary
Keywords: annotation,cli,gilabs,lerobot,robotics
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx<0.30,>=0.27.0
Requires-Dist: rich<15,>=13.0
Requires-Dist: typer[all]<1,>=0.9.0
Description-Content-Type: text/markdown

# gidata CLI Guide

`gidata` is a command-line tool for uploading **and downloading** robotics data on the GI Labs Data Platform. It supports uploading LeRobot datasets, video files, trajectory data, and downloading complete task bundles in **7 formats** — annotation containers (Parquet/JSON/RLDS) or domain-standard recording layouts (LeRobot v3.0, HDF5/ALOHA, ROS 2 Rosbag, UMI passthrough) — for offline analysis, CI pipelines, or ML training. The full format guide lives in [`docs/user-guides/exporting-data.md`](../docs/user-guides/exporting-data.md).

## Installation

Requires Python 3.11+.

```bash
# Recommended (isolated CLI install):
pipx install gidata

# Or with uv:
uv tool install gidata

# Or plain pip into the active environment:
pip install gidata
```

Verify the installation:

```bash
gidata --version
```

Upgrade later with the built-in self-upgrade command — it autodetects
whether you installed via `pipx`, `uv tool`, or plain `pip` and runs the
matching upgrade for you:

```bash
gidata upgrade            # interactive: prompts before installing
gidata upgrade --yes      # skip the prompt
gidata upgrade --check    # report whether an update is available, no install
```

If you prefer to drive the package manager yourself: `pipx upgrade gidata`,
`uv tool upgrade gidata`, or `pip install --upgrade gidata`.

### Local development

Working from a clone of this repo? Install the editable copy instead:

```bash
# From the repository root
pip install -e ./cli
# Or: uv pip install -e ./cli
```

### Releasing a new version

Releases are cut from a `cli-vX.Y.Z` git tag. The `Publish gidata CLI` workflow
(`.github/workflows/cli-release.yml`) builds sdist + wheel with `uv build` and
pushes them to PyPI via [Trusted Publishing](https://docs.pypi.org/trusted-publishers/) — no API tokens are stored in GitHub.

```bash
# 1. Bump the version in cli/pyproject.toml AND cli/gidata/__init__.py
# 2. Commit and push to main
# 3. Tag and push:
git tag cli-v0.1.1
git push origin cli-v0.1.1
```

The workflow guards against tag/version drift and runs `ruff` + `pytest` before
publishing. For dry runs, trigger the workflow manually with `target=testpypi`.

## Authentication

Before uploading, you need to authenticate. There are two methods:

### API Key (Recommended)

Generate an API key from the web UI at **Settings > Account > API Keys**, then:

```bash
gidata auth login --api-key YOUR_KEY --org-id YOUR_ORG_UUID
```

`--url` is optional and defaults to the GILabs Data Center backend, so you normally don't need it. Pass `--url` only to target a local or self-hosted backend (e.g. `--url http://localhost:8000`); it's saved for future commands.

### Where to find your organization ID

Most CLI commands need an org id. Open the dashboard, click the **organization name** in the top-left of the sidebar to expand the switcher — the active org's UUID appears as a small monospace string under the member count. Click it to copy.

You can pass the UUID inline (`--org-id YOUR_ORG_UUID` on `auth login`) or set it independently afterwards:

```bash
gidata config set org-id YOUR_ORG_UUID    # one-time
gidata config show                         # verify the active settings
```

### Email / Password

You can also log in with your Supabase email/password credentials:

```bash
gidata auth login \
  --url http://localhost:8000 \
  --supabase-url http://localhost:54321 \
  --supabase-anon-key YOUR_ANON_KEY
```

You'll be prompted for your email and password.

### Check Status

```bash
gidata auth status
```

Shows the current auth method, API URL, and user info.

### Log Out

```bash
gidata auth logout
```

Removes stored credentials from `~/.config/gidata/`.

## Uploading Data

All upload commands require authentication. Run `gidata auth login` first.

### Upload a LeRobot Dataset

Upload a `.zip` file containing a LeRobot-formatted dataset to a task:

```bash
gidata upload dataset ./my_dataset.zip --task-id TASK_ID
```

The zip should follow the LeRobot v3.0 format:

```
dataset/
├── meta/
│   ├── info.json
│   ├── episodes.jsonl
│   └── tasks.jsonl
├── data/
│   └── episode_{idx}/
│       └── trajectory.parquet
└── videos/
    └── {camera_key}/
        └── episode_{idx}/
            └── file_{idx}.mp4
```

### Upload Videos

Upload video files for a specific episode:

```bash
# Single video
gidata upload videos ./video.mp4 --episode-id EPISODE_ID

# Directory of videos
gidata upload videos ./camera_feeds/ --episode-id EPISODE_ID
```

Supported formats: `.mp4`, `.avi`, `.mov`, `.mkv`, `.webm`.

### Upload Trajectory Data

Upload a `.parquet` trajectory file for an episode:

```bash
gidata upload trajectory ./trajectory.parquet --episode-id EPISODE_ID
```

### Upload a `.umi` Episode

Upload a raw `.umi` recording (the format produced by Quest devices) to a task. The task must be in `data_collection` mode. The CLI opens a chunked upload session against `/api/v1/upload/umi/sessions`, streams the file in 2 MB chunks with `Content-Range` for resume support, then PUTs metadata and polls until the backend finishes converting to LeRobot format (demux → MP4 + trajectory parquet → Episode row).

```bash
# Minimal: infers episode name from the file stem, device_id defaults to "cli"
gidata upload umi ./episode_0.umi --task-id TASK_ID

# With a custom episode name and explicit device identifier
gidata upload umi ./episode_0.umi \
  --task-id TASK_ID \
  --episode-name pickup_run_42 \
  --device-id lab-quest-01

# Point at an explicit metadata.json (otherwise a sibling metadata.json next to the
# .umi file is used automatically, falling back to a minimal {task_id, episode_name} dict)
gidata upload umi ./episode_0.umi --task-id TASK_ID --metadata ./meta.json

# Fire-and-forget: return as soon as bytes are uploaded, don't wait for processing
gidata upload umi ./episode_0.umi --task-id TASK_ID --no-wait
```

By default the command waits up to 30 minutes for the server to finish processing and prints the resulting Episode ID on success. With `--no-wait` it exits immediately after metadata is submitted and prints the session ID so you can check status later.

Constraints enforced by the server:
- Task must exist in the current org and be `data_collection` mode
- Per-device/episode-name uniqueness for active (non-terminal) sessions
- File size capped by `MAX_UMI_FILE_SIZE`

### Upload an Ego Episode (`.tar` or raw rig `.umi`)

Upload an ego payload to an `ego_collection` task. Two payload types are accepted:

- a phone-recorded **`.tar`** (produced by the ego-recorder phone app — H.265 video + IMU JSONL + camera_info JSON + metadata.json bundled together); or
- a raw **RV1106 ego-rig `.umi`** straight off the device. The server validates that it is an ego recording (an `EQ_EGO` camera, no pose stream — a gripper `.umi` is refused and pointed at `upload umi`) and repacks it into the ego layout before ingest.

Either way the flow mirrors UMI exactly: chunked session, 2 MiB chunks with `Content-Range`, server-side ingest into MCAP + Episode row.

```bash
# Phone .tar — episode name from file stem, device_id "cli"
gidata upload ego ./episode_0.tar --task-id TASK_ID

# Raw rig .umi — keep the rig's sibling metadata.json next to it (auto-discovered);
# the server translates that UMI-native blob into the fields it needs.
gidata upload ego ./session_00002/part_000.umi --task-id TASK_ID

# With explicit metadata
gidata upload ego ./episode_0.tar \
  --task-id TASK_ID \
  --episode-name kitchen_pour_01 \
  --device-id ego-phone-3 \
  --metadata ./episode_0.metadata.json
```

Constraints enforced by the server:
- Task must be `ego_collection` mode
- For a `.umi`: must be an ego-rig recording (not a gripper) — content-checked, not by extension
- Payload size ≤ `MAX_EGO_FILE_SIZE` (25 GiB by default — accommodates Pure Ego TARs that carry the raw RV1106 `source/episode.umi` alongside the demuxed artifacts)
- Per-device/episode-name uniqueness for active sessions

## Downloading Task Data

The `gidata download` subcommands pull a task's uploaded data from the backend and write it to your local disk. Useful for running algorithms offline, training models, or archiving completed tasks.

### List Available Tasks

Browse tasks in your organization to find the one you want to pull:

```bash
gidata download list                          # first 50 tasks
gidata download list --search "pickup"        # substring match on task name
gidata download list --status labeled         # filter by status
gidata download list --project-id PROJECT_ID  # filter by project
```

Output shows task id (truncated), name, status, project, and `verified / labeled / collected` progress counts.

### Download a Task

```bash
# By name (resolved via substring search)
gidata download task "aloha_pickup"

# By UUID (exact)
gidata download task 550e8400-e29b-41d4-a716-446655440000
```

If a name matches multiple tasks, the CLI prints a disambiguation table and asks for the exact task id.

**Default behavior** — the CLI:
1. Asks the backend to bundle the task into a zip (annotations + trajectories + episode metadata + signed video URL manifest)
2. Streams the zip with a progress bar
3. Extracts it into `./<task-name>_<short-id>/`
4. Prints a summary (episode count, annotation count, size)

### Downsampling with `--sample-rate`

For large trajectories you often don't need every frame. Downsample on the server before downloading:

```bash
# Keep every other frame (halves the trajectory row count)
gidata download task my_task --sample-rate 0.5

# Keep every 10th frame
gidata download task my_task --sample-rate 0.1

# Equivalent using explicit integer stride
gidata download task my_task --frame-stride 10
```

`--sample-rate R` in `(0, 1]` is converted to `frame_stride = round(1 / R)`. Use `--frame-stride N` directly if you prefer integer semantics (takes precedence if both are set).

### Sampling Episodes

If a task has far more episodes than you need, pick a subset with one of the three **mutually exclusive** flags below. They filter episodes *before* packing — annotations and trajectories for unselected episodes are not downloaded. All three are orthogonal to `--frame-stride` / `--sample-rate` and can be combined with them.

```bash
# Every 3rd episode (ordered by episode_index)
gidata download task my_task --episode-stride 3

# Just the most recent 5 episodes
gidata download task my_task --last-episodes 5

# A representative sample of at most 10 episodes, evenly spaced across the task
gidata download task my_task --max-episodes 10
```

Pick only one — the CLI errors out if you pass more than one. The summary line after download shows `N/M episodes` so you can see how much of the task was sampled.

### Format, Trajectory, and Output Options

```bash
# Annotation containers (legacy)
gidata download task my_task --format parquet           # default
gidata download task my_task --format json
gidata download task my_task --format rlds              # for TFDS / RLDS pipelines

# Recording formats (the data, not just the annotations)
gidata download task my_task --format lerobot           # HuggingFace LeRobot v3.0 dir tree
gidata download task my_task --format rosbag            # ROS 2 .db3 per episode (refs mode)
gidata download task my_task --format umi               # passthrough of original .umi files

# Async formats — CLI polls until the bundle is ready
gidata download task my_task --format hdf5              # ALOHA-style .h5 per episode
gidata download task my_task --format rosbag --inline-videos
gidata download task my_task --format hdf5 --inline-videos

# Skip trajectory parquets (metadata + annotations only)
gidata download task my_task --no-trajectory

# Custom output directory
gidata download task my_task --out /data/offline/my_task

# Keep the raw zip alongside the extracted tree
gidata download task my_task --keep-zip
```

The CLI runs a **pre-flight availability check** before submitting. If the format isn't applicable to the task (e.g. UMI on a LeRobot-only task) it hard-fails with the server's reason rather than silently producing an empty bundle:

```
$ gidata download task my-lerobot-task -f umi
'umi' export not available: UMI export requires UMI-collected episodes
with a stored raw .umi file; this task has none.
```

### Bulk-download all passed / failed QC episodes

`gidata download qc` downloads **every** QC-passed (or QC-failed) episode across your whole organization, divided into per-task folders. Unlike the web QC page's **Export** button (which bundles your checkbox selection, or the whole filtered set capped at 2000 episodes), the CLI has **no cap** — it lists the episodes, groups them by task, and downloads each task's bundle one at a time, so it scales to any size.

```bash
# All passed episodes, default parquet container
gidata download qc --status passed

# All failed episodes as LeRobot, into a chosen directory
gidata download qc --status failed --format lerobot --out ./qc_failed

# Limit to tasks matching a search (name / canonical id / id)
gidata download qc --status passed --search "ballsincontainer"
```

Options: `--status passed|failed` (required), `--format` (parquet | json | rlds | lerobot | umi | rosbag — same sync formats as `download task`; async HDF5 / inline-Rosbag aren't offered here), `--search`, `--no-trajectory`, `--frame-stride`, `--out` (default `./qc_<status>/`), `--force`. Output layout:

```
qc_passed/
└── tasks/
    ├── <task-name>_<id8>/   # that task's passed episodes, in the chosen format
    └── …
```

Requires the **reviewer** role (same gate as the web QC pages). Videos are referenced via signed URLs (manifest), matching `download task`.

### Bundle Layouts

#### Annotation containers (`parquet` / `json` / `rlds`) — default
```
my_task_550e8400/
├── task.json                          # task + project metadata, progress, export params
├── annotations.parquet                # (or .json / .rlds.json depending on --format)
├── episodes/
│   ├── episode_0/
│   │   ├── metadata.json              # fps, length, duration, cameras, storage paths
│   │   └── trajectory.parquet         # downsampled per --sample-rate / --frame-stride
│   └── episode_1/
│       └── ...
└── videos_manifest.json               # signed MP4 URLs (valid 24h), per camera
```

#### LeRobot v3.0 (`--format lerobot`)
```
my_task_550e8400/
├── task.json
├── meta/info.json                     # robot_type, fps, features (cameras)
├── meta/episodes.jsonl                # one row per episode {episode_index, tasks, length}
├── meta/tasks.jsonl                   # action descriptions from annotations
├── data/chunk-000/episode_<idx>.parquet  # trajectory + task_index column
├── videos_manifest.json
└── skipped_episodes.json              # ego (mcap) eps that LeRobot can't carry
```
Round-trip-ingestible by the platform's own `LeRobotIngestor`.

#### HDF5 / ALOHA (`--format hdf5`, async)
```
my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0.h5                   # /observations/qpos, /action, /timestamps, /annotations/segments
│   └── episode_1.h5                   # plus /observations/images/<cam> in --inline-videos mode
└── videos_manifest.json
```

#### ROS 2 Rosbag (`--format rosbag`)
```
my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0/
│   │   ├── episode_0.db3              # ROS 2 sqlite3 bag
│   │   └── metadata.yaml              # rosbag2 sidecar
│   └── episode_1/...
└── videos_manifest.json
```
Topics: `/joint_states`, `/imu` (ego only), `/annotations`, `/video_refs`. With `--inline-videos`, also `/<camera>/image_raw/compressed` (JPEG per frame). Replay with `ros2 bag play episode_0/`.

#### UMI (`--format umi`)
```
my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0.umi                  # byte-copy of raw_umi_path
│   └── episode_0.annotations.json     # per-episode annotations (when present)
└── missing_episodes.json              # episodes without a raw .umi
```

In refs mode (default), videos are **not inlined** — the zip contains `videos_manifest.json` with signed URLs you can fetch separately (e.g. with `curl` or `wget`). This keeps bundle sizes small. Pass `--inline-videos` for HDF5 / Rosbag to embed frames; that path runs as a background job (CLI polls automatically).

## Configuration

Credentials and settings are stored in `~/.config/gidata/`:

| File | Contents |
|------|----------|
| `config.json` | API URL and active organization id |
| `credentials.json` | API key or bearer token + (for email/password logins) supabase URL/anon key for auto-refresh (file permissions: `600`) |

Inspect or update settings via the `config` subcommand instead of editing the JSON by hand:

```bash
gidata config show                                              # print current settings
gidata config set api-url http://localhost:8000                 # override backend URL (default: GILabs Data Center)
gidata config set org-id 550e8400-e29b-41d4-a716-446655440000   # set active org
gidata config unset org-id                                      # clear an existing value
```

### Troubleshooting

**`Permission denied: Not a member of this organization`** on any upload/download — the org id stored in `~/.config/gidata/config.json` is wrong (a stale slug, a typo, or set to a non-UUID like `testtest`). The backend looks up `OrganizationMember` by exact UUID and a non-match returns 403 even though your API key / bearer token verified fine. Fix:

```bash
gidata config show                                  # confirm current org id
gidata config set org-id 550e8400-e29b-41d4-a716-446655440000
```

Find the correct UUID in the dashboard by clicking the organization name in the top-left of the sidebar — the active org's UUID appears as a small monospace string under the member count.

**`gidata auth status` says "Authenticated" but uploads still 403** — `auth status` only verifies the credential against `/users/me` (which is org-agnostic). It does **not** validate the configured org id. A stale or wrong `org-id` will pass `auth status` and fail every org-scoped call.

### Bearer token auto-refresh

Email/password logins persist `supabase_url` and `supabase_anon_key` alongside the access/refresh token pair. When a request returns 401 the CLI calls Supabase's `/auth/v1/token?grant_type=refresh_token` once, swaps in the new access token, and retries the original request — so a long-running shell stays authenticated past the access-token TTL without re-prompting. Streamed multipart uploads can't be replayed transparently; if a 401 lands mid-upload you'll need to re-run the command. API-key logins (the recommended path) never expire and skip the refresh path entirely.

## Command Reference

```
gidata --version                              Show version
gidata auth login --api-key KEY [--url URL]   Log in with API key
gidata auth login [--url URL] [--supabase-*]  Log in with email/password (auto-refresh enabled)
gidata auth logout                            Clear stored credentials
gidata auth status                            Show auth state and user info
gidata config show                            Print current settings + file paths
gidata config set KEY VALUE                   Update api-url or org-id
gidata config unset KEY                       Remove a setting
gidata upload dataset PATH --task-id ID       Upload .zip LeRobot dataset (streamed)
gidata upload videos PATH --episode-id ID     Upload video file(s) (streamed)
gidata upload trajectory PATH --episode-id ID Upload .parquet trajectory
gidata upload umi PATH --task-id ID [opts]    Upload .umi episode (chunked, processed server-side)
gidata upload ego PATH --task-id ID [opts]    Upload ego .tar episode (chunked, ego_collection tasks only)
gidata download list [--search TEXT]          List tasks in the current org
gidata download task TASK [options]           Download a task's uploaded data
gidata download qc --status passed|failed     Download all passed/failed QC episodes (by task, uncapped)
gidata firmware publish APK ...               Publish a firmware APK release (platform admin)
```

### `gidata upload umi` options

| Flag | Description | Default |
|------|-------------|---------|
| `--task-id, -t` | Target task ID (must be `data_collection` mode) | _required_ |
| `--device-id, -d` | Source device identifier | `cli` |
| `--episode-name, -n` | Episode name — unique per device for active sessions | file stem |
| `--metadata, -m` | Path to metadata.json | sibling `metadata.json`, else minimal dict |
| `--no-wait` | Return after upload without polling for processing status | off |

### `gidata upload ego` options

Same flags as `upload umi`, but the task must be `ego_collection` mode and the file must be `.tar`.

### `gidata download task` options

| Flag | Description | Default |
|------|-------------|---------|
| `--format, -f` | Export format: `parquet`, `json`, `rlds` (annotation containers) or `lerobot`, `hdf5`, `rosbag`, `umi` (recording formats). HDF5 + inline-mode Rosbag run as async jobs that the CLI polls automatically. | `parquet` |
| `--inline-videos / --no-inline-videos` | Embed video frames inside the bundle (HDF5/Rosbag only). Default off — videos referenced via signed URLs in `videos_manifest.json`. Enabling this switches to the async export-job path. | `--no-inline-videos` |
| `--sample-rate, -r` | Keep fraction of frames in `(0, 1]` — converted to integer stride | `1.0` (all frames) |
| `--frame-stride, -S` | Explicit integer stride; takes precedence over `--sample-rate` | `1` |
| `--episode-stride` | Keep every Nth episode. Mutually exclusive with the two below. | off (all episodes) |
| `--last-episodes` | Keep only the last N episodes. Mutually exclusive with the two others. | off |
| `--max-episodes` | Cap at N episodes, evenly spaced. Mutually exclusive with the two others. | off |
| `--no-trajectory` | Skip trajectory parquets | off |
| `--out, -o` | Output directory | `./<task-name>_<short-id>/` |
| `--keep-zip` | Keep the downloaded zip file | off |
| `--force, -y` | Overwrite a non-empty `--out` directory without prompting | off |

## Examples

```bash
# Full workflow: authenticate and upload a dataset
gidata auth login --api-key gidata_abc123...
gidata upload dataset ./aloha_pickup.zip --task-id 550e8400-e29b-41d4-a716-446655440000

# Check who you're logged in as
gidata auth status

# Upload videos from a directory
gidata upload videos ./recordings/cam_left/ --episode-id ep-001

# Upload trajectory separately
gidata upload trajectory ./data/episode_0/trajectory.parquet --episode-id ep-001

# List labeled tasks and pull one for offline ML training
gidata download list --status labeled
gidata download task "aloha_pickup_v2" --sample-rate 0.5 --out ./training_data/

# Batch-pull several tasks with a shell loop
for t in pickup stack place; do
  gidata download task "$t" --format parquet --out ./data/$t
done
```
