Metadata-Version: 2.4
Name: snap-cv
Version: 0.1.1
Summary: Modular vision model inference toolkit for rapid prototyping
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: click>=8.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.24
Requires-Dist: opencv-python>=4.8
Requires-Dist: huggingface-hub>=0.20
Requires-Dist: tqdm>=4.65
Requires-Dist: Pillow>=10.0
Provides-Extra: sam2
Requires-Dist: torch>=2.0; extra == "sam2"
Requires-Dist: torchvision>=0.15; extra == "sam2"
Requires-Dist: sam2>=0.4; extra == "sam2"
Provides-Extra: sam3
Requires-Dist: torch>=2.0; extra == "sam3"
Requires-Dist: torchvision>=0.15; extra == "sam3"
Requires-Dist: sam3>=0.1.0; extra == "sam3"
Provides-Extra: yolo
Requires-Dist: ultralytics>=8.2; extra == "yolo"
Provides-Extra: grounding-dino
Requires-Dist: torch>=2.0; extra == "grounding-dino"
Requires-Dist: torchvision>=0.15; extra == "grounding-dino"
Requires-Dist: groundingdino-py>=0.4; extra == "grounding-dino"
Provides-Extra: depth-anything
Requires-Dist: torch>=2.0; extra == "depth-anything"
Requires-Dist: torchvision>=0.15; extra == "depth-anything"
Requires-Dist: transformers>=4.36; extra == "depth-anything"
Provides-Extra: ui
Requires-Dist: gradio>=4.0; extra == "ui"
Provides-Extra: all
Requires-Dist: snap-cv[sam2]; extra == "all"
Requires-Dist: snap-cv[sam3]; extra == "all"
Requires-Dist: snap-cv[yolo]; extra == "all"
Requires-Dist: snap-cv[grounding-dino]; extra == "all"
Requires-Dist: snap-cv[depth-anything]; extra == "all"
Requires-Dist: snap-cv[ui]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"

# SnapVision

> Internal vision toolkit — Solomon 3D

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Quick Start](#quick-start)
- [Installation](#installation)
- [Supported Models](#supported-models)
- [Usage](#usage)
  - [Python API](#python-api)
  - [CLI](#cli)
  - [Using Local Weights (Gated Models)](#using-local-weights-gated-models)
  - [Gradio Web UI](#gradio-web-ui)
- [Results](#results)
- [Adding New Models](#adding-new-models)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [Development](#development)
- [License](#license)

---

## Overview

SnapVision is a modular, pip-installable Python toolkit for rapid prototyping with vision models. It standardizes the **download → load → predict → save** cycle so engineers can go from "I have images" to "results" in minutes, not hours.

Every model follows the same interface: call `snap_vision.load()`, call `.predict()`, call `.save()`. The right weights are fetched automatically from HuggingFace on first use and cached locally. For gated models like SAM3, you can supply local weights instead.

The package is named `snap-cv` on pip and imported as `snap_vision`. The CLI command is `snap-cv`.

---

## Features

- **Modular model support via pip extras** — install only the backends you need; unused models add zero overhead
- **Auto-download with local caching** — weights are fetched from HuggingFace or direct URLs on first use and stored in `~/.snap_vision/weights/`
- **Local weights support for gated models** — point `weights_dir` at your local copy for models behind a HuggingFace access gate (e.g. SAM3)
- **Typed results with built-in visualization and save** — `DetectionResult`, `SegmentationResult`, and `DepthResult` each carry `.save()`, `.to_json()`, and `.visualize()` methods
- **Three interfaces** — Python API, CLI, and Gradio web UI; all share the same underlying adapter layer
- **Cross-platform device auto-detection** — prefers CUDA, falls back to MPS (Apple Silicon), then CPU

---

## Quick Start

```bash
pip install snap-cv[yolo]
```

```python
import cv2
import snap_vision as sv

model = sv.load("yolov8")
image = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)
result = model.predict(image, confidence=0.4)
result.save("./output", image=image)
```

---

## Installation

The core package installs only lightweight dependencies (Click, PyYAML, NumPy, OpenCV, Pillow, HuggingFace Hub). Install model-specific extras only when you need them.

```bash
# Core only (no model backends)
pip install snap-cv

# Object detection with YOLOv8
pip install snap-cv[yolo]

# Segment Anything 2
pip install snap-cv[sam2]

# Segment Anything 3 (gated — see Local Weights section)
pip install snap-cv[sam3]

# Open-set detection with text prompts
pip install snap-cv[grounding-dino]

# Monocular depth estimation
pip install snap-cv[depth-anything]

# Gradio web UI
pip install snap-cv[ui]

# Everything
pip install snap-cv[all]

# Development dependencies
pip install snap-cv[dev]
```

> **Python requirement:** 3.9 or later.

---

## Supported Models

| Model | Task | Extra | Description | Gated |
|---|---|---|---|---|
| `yolov8` | `detection` | `[yolo]` | YOLOv8x by Ultralytics — high-accuracy detection | No |
| `yolov8n` | `detection` | `[yolo]` | YOLOv8 Nano — fast, lightweight detection | No |
| `sam2` | `prompted_segmentation` | `[sam2]` | Segment Anything 2 by Meta | No |
| `sam3` | `prompted_segmentation` | `[sam3]` | Segment Anything 3 with native text prompts | Yes |
| `grounding_dino` | `grounded_detection` | `[grounding-dino]` | Open-set detection driven by text queries | No |
| `depth_anything` | `depth_estimation` | `[depth-anything]` | Depth Anything V2 — monocular depth estimation | No |

Weights are downloaded automatically for non-gated models. See [Using Local Weights](#using-local-weights-gated-models) for gated models.

---

## Usage

### Python API

**Load and predict — detection**

```python
import cv2
import snap_vision as sv

# Load a model (downloads weights on first call, cached afterward)
model = sv.load("yolov8")

image = cv2.cvtColor(cv2.imread("scene.jpg"), cv2.COLOR_BGR2RGB)
result = model.predict(image, confidence=0.35)

print(f"Found {len(result.boxes)} objects")
for box in result.boxes:
    print(f"  {box.label}: {box.score:.2f}  [{box.x1:.0f}, {box.y1:.0f}, {box.x2:.0f}, {box.y2:.0f}]")

# Save result.json + visualization.png under ./output/scene/
result.save("./output/scene", image=image)
```

**Segmentation with SAM2**

```python
model = sv.load("sam2")

# Provide point prompts (foreground clicks)
result = model.predict(image, points=[(320, 240), (400, 200)])

# Or provide a bounding box prompt
result = model.predict(image, boxes=[[100, 80, 500, 400]])

print(f"Segmented {len(result.masks)} mask(s), scores: {result.scores}")
result.save("./output/seg", image=image)
```

**Text-prompted detection with GroundingDINO**

```python
model = sv.load("grounding_dino")
result = model.predict(image, text_prompt="robotic arm . screw . bracket")
result.save("./output/grounded", image=image)
```

**Depth estimation**

```python
model = sv.load("depth_anything")
result = model.predict(image)

import numpy as np
print(f"Depth range: {result.depth_map.min():.2f} – {result.depth_map.max():.2f}")
result.save("./output/depth")  # saves depth_map.npy + colorized visualization.png
```

**Force a specific device**

```python
model = sv.load("sam2", device="cpu")       # force CPU
model = sv.load("yolov8", device="cuda")    # force CUDA GPU 0
```

**Visualize without saving**

```python
vis = model.visualize(image, result)        # returns np.ndarray (H, W, 3)
cv2.imshow("result", vis)
cv2.waitKey(0)
```

---

### CLI

**List all available models**

```bash
snap-cv list
```

```
Model                Task                      Downloaded   Description
---------------------------------------------------------------------------------
yolov8               detection                 yes          YOLOv8 object detection by Ultralytics
yolov8n              detection                 no           YOLOv8 nano — fast, lightweight detection
sam2                 prompted_segmentation     no           Segment Anything 2 by Meta — prompted segmentation
...

Weight cache: /Users/you/.snap_vision/weights
```

Filter by task:

```bash
snap-cv list --task detection
```

**Pre-download weights**

```bash
snap-cv download yolov8
snap-cv download sam2
snap-cv download --all          # download every registered model
snap-cv download yolov8 --force # re-download even if cached
```

**Run inference**

```bash
# Single image, print JSON to stdout
snap-cv predict yolov8 --input photo.jpg

# Single image, save structured output
snap-cv predict yolov8 --input photo.jpg --save-dir ./results

# Entire directory of images
snap-cv predict yolov8 --input ./frames/ --save-dir ./results

# With confidence override and display window
snap-cv predict yolov8 --input photo.jpg --confidence 0.5 --show

# GroundingDINO with a text prompt
snap-cv predict grounding_dino --input photo.jpg \
    --prompt "robotic arm . screw" --save-dir ./results

# SAM3 with local weights (gated model)
snap-cv predict sam3 --input photo.jpg \
    --weights-dir /data/weights/sam3/ --save-dir ./results

# Force a device
snap-cv predict depth_anything --input photo.jpg --device cpu
```

**Remove cached weights**

```bash
snap-cv remove sam2
```

**Show configuration**

```bash
snap-cv config --show
```

```
Weight cache directory: /Users/you/.snap_vision/weights
Set SNAP_VISION_CACHE_DIR env var to override.
```

**Launch the web UI**

```bash
snap-cv ui
snap-cv ui --port 8080
snap-cv ui --model yolov8      # pre-load a specific model
```

---

### Using Local Weights (Gated Models)

SAM3 is hosted on a gated HuggingFace repository (`facebook/sam3`). You must request access through HuggingFace before weights can be downloaded automatically. Two workflows are supported.

**Option A — HuggingFace token (automatic download)**

Once your access request is approved, set your token and let SnapVision handle the download:

```bash
export HF_TOKEN=hf_your_token_here
snap-cv download sam3
```

Subsequent calls require no token because weights are cached locally.

**Option B — local weights directory**

Download the weights yourself through the HuggingFace web interface or `huggingface-cli`, then point SnapVision at that directory:

```bash
# CLI
snap-cv predict sam3 \
    --input photo.jpg \
    --weights-dir /data/weights/sam3/ \
    --save-dir ./results
```

```python
# Python API
model = sv.load("sam3", weights_dir="/data/weights/sam3/")
result = model.predict(image, text_prompt="the circuit board")
```

SnapVision expects to find a `.safetensors`, `.pt`, or `.pth` file inside the `weights_dir`. The directory layout from a standard HuggingFace snapshot download works without modification.

The `--weights-dir` flag works for any model, not only gated ones. Use it to pin a specific model version or to work entirely offline.

---

### Gradio Web UI

The web UI provides a browser-based interface for interactive prototyping. It requires the `[ui]` extra.

```bash
pip install snap-cv[ui]
snap-cv ui
```

Open `http://localhost:7860` in your browser. The UI exposes:

- An image upload panel
- A model selector showing all registered models
- A confidence slider (0.0 – 1.0)
- A text prompt field (used by GroundingDINO and SAM3)
- A "Run Inference" button
- A side-by-side view of the annotated result image and the raw JSON output

Models are loaded on first use and kept in memory for subsequent requests, so switching between images with the same model is fast.

```bash
# Start on a custom port with a pre-loaded model
snap-cv ui --model yolov8 --port 8080
```

---

## Results

Every model returns a typed result object. All result types share three methods:

| Method | Description |
|---|---|
| `.save(path, image=None)` | Write results to a directory; creates it if it does not exist |
| `.to_json()` | Return a JSON string summary of the result |
| `.visualize(image)` | Return an annotated `np.ndarray` (H, W, 3) |

### DetectionResult

Returned by `yolov8`, `yolov8n`, and `grounding_dino`.

```python
result.boxes          # List[Box]
result.boxes[0].x1    # float — left edge in pixels
result.boxes[0].y1    # float — top edge in pixels
result.boxes[0].x2    # float — right edge in pixels
result.boxes[0].y2    # float — bottom edge in pixels
result.boxes[0].label # str — class name or phrase
result.boxes[0].score # float — confidence score
result.boxes[0].width  # property: x2 - x1
result.boxes[0].height # property: y2 - y1
```

**Save directory layout:**

```
output/
  result.json          # serialized boxes with labels and scores
  visualization.png    # image with bounding boxes drawn
  metadata.json        # model name, confidence threshold, etc.
```

### SegmentationResult

Returned by `sam2` and `sam3`.

```python
result.masks      # List[np.ndarray] — each mask is (H, W) uint8, values 0 or 1
result.scores     # List[float] — confidence per mask
result.labels     # List[str] — label per mask (may be empty)
```

**Save directory layout:**

```
output/
  result.json          # mask count, scores, shapes
  mask_0.png           # first mask as grayscale PNG (0 or 255)
  mask_1.png           # second mask, and so on
  visualization.png    # masks overlaid on image with semi-transparent colors
  metadata.json        # model name, prompts used
```

### DepthResult

Returned by `depth_anything`.

```python
result.depth_map   # np.ndarray (H, W) float32 — relative depth values
```

**Save directory layout:**

```
output/
  result.json          # shape, min_depth, max_depth
  depth_map.npy        # raw depth array in NumPy format
  visualization.png    # depth map colorized with the INFERNO colormap
  metadata.json        # model name
```

---

## Adding New Models

Adding a model requires three steps: writing an adapter class, registering the model in `model_registry.yaml`, and declaring any new pip dependencies in `pyproject.toml`.

### Step 1 — Write an adapter class

Create a file under `src/snap_vision/models/`. Extend the appropriate base class:

| Task | Base class |
|---|---|
| Detection | `DetectionModel` |
| Prompted segmentation | `PromptedSegmentationModel` |
| Text-prompted detection | `GroundedDetectionModel` |
| Depth estimation | `DepthEstimationModel` |

Implement three methods: `load`, `predict`, and (if the base does not provide one) `visualize`.

```python
# src/snap_vision/models/my_model.py

from pathlib import Path
from typing import Any
import numpy as np

from snap_vision.core.base import DetectionModel
from snap_vision.core.results import Box, DetectionResult


class MyModelAdapter(DetectionModel):
    """Adapter for MyModel."""

    DEPS = ["my_model_package"]   # pip package(s) that must be importable

    def load(self, weights_dir: Path, device: str) -> None:
        from my_model_package import MyModel

        weight_files = sorted(weights_dir.glob("*.pt"))
        if not weight_files:
            raise FileNotFoundError(f"No .pt file found in {weights_dir}")

        self.model = MyModel(str(weight_files[0])).to(device)
        self.device = device

    def predict(
        self,
        image: np.ndarray,
        confidence: float = 0.25,
        **kwargs: Any,
    ) -> DetectionResult:
        raw = self.model.infer(image, conf=confidence)

        boxes = [
            Box(x1=d["x1"], y1=d["y1"], x2=d["x2"], y2=d["y2"],
                label=d["label"], score=d["score"])
            for d in raw["detections"]
        ]
        return DetectionResult(
            boxes=boxes,
            metadata={"model": "my_model", "confidence_threshold": confidence},
        )
```

The `check_deps()` method on `SnapModel` automatically checks `DEPS` before loading and raises a helpful `ImportError` pointing to the correct pip extra.

### Step 2 — Register the model

Add an entry to `src/snap_vision/model_registry.yaml`:

```yaml
models:
  my_model:
    task: detection
    adapter: snap_vision.models.my_model.MyModelAdapter
    deps: [my_model_package]
    description: "MyModel — short description for snap-cv list"
    weights:
      source: huggingface          # or "url"
      repo_id: org/my-model-repo
      files: [model.pt]
      requires_token: false
    input_schema:
      image: required
      confidence: optional
    default_params:
      confidence: 0.25
```

For direct URL downloads, use `source: url`, `url: https://...`, and `filename: model.pt` instead of `repo_id` and `files`.

### Step 3 — Add the pip extra

In `pyproject.toml`, add an entry under `[project.optional-dependencies]` and add it to `all`:

```toml
[project.optional-dependencies]
my-model = ["my_model_package>=1.0"]
all = [
    "snap-cv[sam2]",
    "snap-cv[yolo]",
    # ...
    "snap-cv[my-model]",
]
```

After these three steps, `snap_vision.load("my_model")` and `snap-cv predict my_model ...` work without any further changes.

---

## Architecture

```
src/snap_vision/
  __init__.py              # exposes sv.load, DetectionResult, SegmentationResult, DepthResult
  cli.py                   # Click command group: list, download, predict, remove, config, ui
  model_registry.yaml      # single source of truth for all model metadata
  core/
    base.py                # SnapModel ABC and task-specific base classes
    device.py              # CUDA / MPS / CPU auto-detection
    hub.py                 # ModelHub.load() — the main user-facing factory
    registry.py            # ModelRegistry — reads model_registry.yaml into ModelInfo dataclasses
    results.py             # DetectionResult, SegmentationResult, DepthResult, Box
    weights.py             # download_weights, is_downloaded, remove_weights, get_cache_dir
  models/
    yolo.py                # YOLOAdapter (Ultralytics)
    sam2.py                # SAM2Adapter (Meta)
    sam3.py                # SAM3Adapter (Meta, gated)
    grounding_dino.py      # GroundingDINOAdapter (IDEA Research)
    depth_anything.py      # DepthAnythingAdapter (via HuggingFace transformers pipeline)
  ui/
    app.py                 # Gradio Blocks interface, launch_ui()
  mcp/
    __init__.py            # (reserved)
```

**Key design decisions:**

- **Registry-driven** — all model metadata lives in `model_registry.yaml`. Adding a model never requires changing Python source outside of the adapter file itself.
- **Lazy imports** — model backends (`ultralytics`, `sam2`, `transformers`, etc.) are imported inside `load()` and `predict()`, so the core package imports cleanly even when extras are absent. A missing dependency produces a clear error pointing to the correct `pip install` command.
- **Adapter pattern** — each model file is self-contained. Swapping or upgrading a backend only touches one file.
- **Typed results** — rather than returning raw dicts or framework tensors, every model returns a structured result object. This makes downstream code framework-agnostic and provides consistent save/visualize behavior.
- **Cache-first weights** — `download_weights()` is a no-op when weights already exist locally. The `--force` flag bypasses this for explicit re-downloads.

---

## Configuration

### Cache directory

By default, weights are stored at `~/.snap_vision/weights/<model_name>/`. Override this with an environment variable:

```bash
export SNAP_VISION_CACHE_DIR=/mnt/ssd/snap_vision_weights
```

Check the active cache directory:

```bash
snap-cv config --show
```

### HuggingFace token

Required for gated models (currently SAM3). Set the standard HuggingFace environment variable:

```bash
export HF_TOKEN=hf_your_token_here
```

SnapVision reads `HF_TOKEN` automatically when downloading a model whose registry entry has `requires_token: true`. The token is never written to disk.

### Summary

| Variable | Purpose | Default |
|---|---|---|
| `SNAP_VISION_CACHE_DIR` | Root directory for cached weights | `~/.snap_vision/weights/` |
| `HF_TOKEN` | HuggingFace access token for gated models | (not set) |

---

## Development

```bash
git clone <repo-url>
cd common_tools

# Editable install with dev dependencies
pip install -e ".[dev]"

# Run all tests
pytest

# Run tests with coverage report
pytest --cov=snap_vision --cov-report=term-missing
```

Tests live in `tests/` and cover the core layer (registry, weights, device detection, base classes, and all result types) without requiring any model backend to be installed.

To add tests for a new model adapter, follow the patterns in `tests/test_base.py`: use a minimal mock subclass that sets `DEPS = []` so tests run without the real backend package.

---

## License

MIT. See `LICENSE` for the full text.

---

*SnapVision is an internal tool maintained by the Solomon 3D vision and robotics team.*
