Metadata-Version: 2.4
Name: rgbd-pd-viewer
Version: 0.1.0
Summary: Open3D GUI viewer for RGB-D frames with point picking and Robot Base Marker for camera-to-world calibration.
Author-email: Yuzhen Chen <yuzhenchen@g.harvard.edu>
License: MIT
Project-URL: Homepage, https://github.com/yuzhenchen/rgbd-pd-viewer
Project-URL: Issues, https://github.com/yuzhenchen/rgbd-pd-viewer/issues
Keywords: rgbd,point-cloud,open3d,depth,visualization,robotics,calibration
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Graphics :: Viewers
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20
Requires-Dist: pillow>=8.0
Requires-Dist: open3d>=0.17
Dynamic: license-file

# rgbd-pd-viewer

An interactive **Open3D GUI** viewer for RGB-D data. Loads a single
(depth, RGB) pair or a synchronized directory of pairs, back-projects them
into a coloured point cloud, and exposes:

- frame navigation (Prev / Next / jump-to-frame)
- camera controls (azimuth / elevation / distance sliders + free mouse)
- **point picking** — Shift + left-click to read camera-frame XYZ
- **Robot Base Marker** — place a sphere at the picked point, nudge it
  onto the robot base, rotate the world frame, and save the resulting
  `T_wc` / `T_cw` to JSON (handy for hand-eye / camera-to-world calibration)
- a camera-origin frustum so the data-camera location is always visible
- adding extra (depth, RGB) sources on the fly, each with its own
  affine depth correction `depth_new = A * depth + B`

The viewer assumes inputs land at **480 × 640** after a centre-crop +
resize; if your raw depth/RGB has a different aspect ratio the loader
crops to match (never stretches), then resizes.

---

## Installation

```bash
pip install rgbd-pd-viewer
```

Or, from a checkout:

```bash
git clone https://github.com/yuzhenchen/rgbd-pd-viewer.git
cd rgbd-pd-viewer
pip install -e .
```

> **Note on Open3D.** On Linux the Open3D GUI needs an OpenGL-capable
> display. Over SSH use `ssh -X` (or VirtualGL); inside Docker you'll
> need to forward `$DISPLAY` and mount `/tmp/.X11-unix`.

---

## Quick start

There are two ways to load data.

### Mode 1 — pass everything on the command line (single frame)

Positional order is **fixed**: `RGB DEPTH [INTRINSICS]`.

```bash
rgbd-pd-viewer color.png depth.npy 640,640,320,240
```

- `RGB`        — path to `.jpg / .png / .bmp / .tiff`
- `DEPTH`      — path to `.npy / .npz`
- `INTRINSICS` — *optional*, four comma-separated floats `fx,fy,cx,cy`
  (in pixels, in that exact order). **No spaces.** Default if omitted:
  `640,640,320,240`.

So the shortest form is just:

```bash
rgbd-pd-viewer color.png depth.npy
```

### Mode 2 — launch the GUI empty and pick files inside

```bash
rgbd-pd-viewer
```

The viewer opens with an empty scene. Use the sidebar:

- **Load Primary RGBD…** — two-step file dialog: pick depth first, then RGB.
- **Camera Intrinsics** — edit `fx / fy / cx / cy` and click **Apply
  Intrinsics**; the current point cloud (and any extra sources) is
  re-projected immediately.
- **+ Add Source (depth + RGB)** — overlay additional point clouds on
  the same scene.

You can also pre-fill intrinsics so you don't have to type them in the UI:

```bash
rgbd-pd-viewer --intrinsics 640,640,320,240
```

### Directory of frames (multi-frame sequence)

Pairs are matched by stripping `--depth-suffix` from each depth filename
and looking for `<stem><rgb-suffix>` in `--rgb-dir`.

```bash
rgbd-pd-viewer \
  --depth-dir    /path/to/depth_npy \
  --rgb-dir      /path/to/color_jpg \
  --intrinsics   640,640,320,240 \
  --depth-suffix _480_640.npy \
  --rgb-suffix   .jpg
```

### Pre-loading a previous calibration

```bash
rgbd-pd-viewer color.png depth.npy --t-wc-init T_wc_real.json
```

The JSON is expected to contain either a flat `robot_base_in_camera_xyz`
field or a 4 × 4 `T_cw` matrix; both forms are written by the viewer's
own "Save T_wc to JSON…" button.

---

## All CLI flags

```
positional (single-frame mode)
  RGB                        path to RGB image  (.jpg / .png / .bmp / .tiff)
  DEPTH                      path to depth file (.npy / .npz)
  INTRINSICS                 fx,fy,cx,cy (4 floats, comma-separated, pixels)
                             default: 640,640,320,240

flags
  --intrinsics fx,fy,cx,cy   same format as the positional, useful with
                             directory mode or when launching an empty UI

directory mode (multi-frame)
  --depth-dir DIR            directory of depth files
  --rgb-dir DIR              directory of RGB images
  --depth-suffix STR         match depth files by suffix (default _480_640.npy)
  --rgb-suffix STR           match RGB files by suffix (default .jpg)

rendering
  --depth-scale FLOAT        multiplier on raw depth (default 1.0;
                             RealSense uint16 mm → use 0.001)
  --max-depth FLOAT          clip Z >= this in metres (default 10.0)
  --max-points INT           uniform-stride subsample cap (default: no cap)

optional calibration
  --t-wc-init PATH           JSON from a previous "Save T_wc" — pre-populates
                             the Robot Base Marker on launch
```

---

## On-screen controls

| Action                           | Input                                    |
| -------------------------------- | ---------------------------------------- |
| Rotate view                      | Left-drag, or Azimuth / Elevation sliders|
| Zoom                             | Scroll wheel, or Distance slider         |
| Pan                              | Right-drag                               |
| Prev / Next frame                | `<  Prev` / `Next  >` buttons            |
| Jump to specific frame           | Type N, click **Go**                     |
| Toggle world axes                | **Grid: OFF / ON**                       |
| Toggle camera frustum            | **Camera Marker: ON / OFF**              |
| Pick a 3-D point                 | **Shift + Left-click**                   |
| Drop sphere at the last pick     | **Place sphere at last pick**            |
| Nudge sphere along world X/Y/Z   | `X-` / `X+` / `Y-` / `Y+` / `Z-` / `Z+`  |
| Rotate world frame Rx / Ry / Rz  | `Rx-` / `Rx+` etc. (step in degrees)     |
| Save calibration                 | **Save T_wc to JSON…**                   |
| Add another (depth, RGB) source  | **+ Add Source (depth + RGB)**           |
| Affine depth correction          | edit `A`, `B`, click **Apply Scale**     |

---

## Library use

The functions and the GUI class are also importable:

```python
from rgbd_pd_viewer import (
    PointCloudApp,
    build_o3d_pointcloud,
    crop_and_resize,
    load_frame,
    get_sorted_frame_pairs,
)

# 1) Just back-project a depth+RGB pair to an Open3D PointCloud:
depth, rgb = load_frame("depth.npy", "color.png")
pcd = build_o3d_pointcloud(
    depth, rgb,
    fx=640, fy=640, cx=320, cy=240,
    depth_scale=1.0, max_depth=10.0,
)

# 2) Or launch the full GUI on a list of pairs:
pairs = get_sorted_frame_pairs("depth_dir", "rgb_dir",
                               depth_suffix="_480_640.npy",
                               rgb_suffix=".jpg")
app = PointCloudApp(pairs, fx=640, fy=640, cx=320, cy=240)
app.run()
```

### Supported depth formats

| Suffix          | Loaded as                                     |
| --------------- | --------------------------------------------- |
| `.npy`          | `float32` array; if 3-D, channel 0 is taken   |
| `.npz`          | key `"depth"` if present, else `npz.files[0]` |
| `.png`/`.jpg`/… | RGB image (`uint8`, 3 channels)               |

Depth values are interpreted in **metres** after multiplication by
`--depth-scale`. For RealSense raw `uint16` saved as `.npy`, you typically
want `--depth-scale 0.001`.

### Robot Base Marker — JSON schema

Saving writes:

```json
{
  "robot_base_in_camera_xyz": [x, y, z],
  "T_cw": [[...], [...], [...], [...]],
  "T_wc": [[...], [...], [...], [...]],
  "R_cw": [[...], [...], [...]],
  "rotation": "identity (only translation set) | manually adjusted via Rx/Ry/Rz buttons",
  "convention": "World origin = robot base. T_cw maps world->camera; T_cw[:3,:3] = R_cw (cols = world axes in camera frame); T_cw[:3,3] = robot-base position in camera frame. T_wc = inv(T_cw).",
  "source": "manual alignment via rgbd_pd_viewer Robot Base Marker"
}
```

---

## Building & publishing

```bash
pip install build twine
python -m build                 # creates dist/*.whl and dist/*.tar.gz
twine check dist/*
twine upload dist/*             # or: twine upload -r testpypi dist/*
```

---

## License

[MIT](LICENSE) © Yuzhen Chen
