Metadata-Version: 2.4
Name: vd3
Version: 1.2.0
Summary: DVC-backed media database for computer vision and ML developers
Project-URL: Homepage, https://github.com/muncasterconsulting/vd3
Project-URL: Repository, https://github.com/muncasterconsulting/vd3
Project-URL: Issues, https://github.com/muncasterconsulting/vd3/issues
Author-email: Justin Muncaster <justin@muncasterconsulting.com>, Alec Wicklund <alec.wicklund@muncasterconsulting.com>, Paul Filitchkin <paul.filitchkin@muncasterconsulting.com>
License-Expression: MIT
Keywords: computer-vision,dataset,dvc,video
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.1.0
Requires-Dist: dvc-azure>=3.0
Requires-Dist: dvc-gdrive>=3.0
Requires-Dist: dvc-gs>=3.0
Requires-Dist: dvc-s3>=3.0
Requires-Dist: dvc>=3.50.0
Requires-Dist: ffmpeg-python>=0.2.0
Requires-Dist: orjson>=3.10.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Requires-Dist: watchdog>=4.0
Description-Content-Type: text/markdown

# VD3


A [DVC](https://dvc.org)-backed media database for computer vision and ML developers. Tracks video and imageset assets, annotations, and worksets as MP4/JSON media with CSV-based metadata, so datasets stay versioned and reproducible across local disks and remote storage backends.

## Contents

- [Installation](#installation)
- [Quick Start](#quick-start)
- [Core Concepts](#core-concepts)
- [Adding Assets](#adding-assets)
  - [Videos](#videos)
  - [Imagesets](#imagesets)
  - [Annotation layers](#annotation-layers)
  - [Layer groups](#layer-groups)
  - [Import COCO Annotations](#import-coco-annotations)
  - [Git-backed vs. DVC-backed annotations](#git-backed-vs-dvc-backed-annotations)
  - [Local (uncommitted) layers](#local-uncommitted-layers)
  - [Layer versioning (status, push, pull, revert)](#layer-versioning-status-push-pull-revert)
- [Worksets](#worksets)
- [Datasources](#datasources)
- [Remote Storage](#remote-storage)
- [Override DB Path](#override-db-path)
- [Listing & Inspection](#listing--inspection)
- [Health Checks](#health-checks)
- [Exporting](#exporting)
- [Library API](#library-api)
- [Versioning & stability](#versioning--stability)
  - [Public API](#public-api)
  - [Deprecations](#deprecations)
- [CLI Reference](#cli-reference)
- [Misc](#misc)
  - [Asset and datasource names](#asset-and-datasource-names)

## Installation

```bash
pip install vd3
```
With **uv** run:
```bash
uv pip install vd3
```

Add as a dependency to your project:

```toml
# pyproject.toml
[project]
dependencies = ["vd3"]
```
Then resolve dependencies and sync:
```bash
uv sync
```

## Quick Start

```bash
# Initialize a content database in the current directory
vd3 db init

# ...or in a specific directory
vd3 db init /path/to/mydb

# Add a video under a datasource
vd3 datasource add-video my-datasource clip.mp4 -p /path/to/mydb

# Add multiple videos with a glob (quote to prevent shell expansion)
vd3 datasource add-video my-datasource '*.mp4' -p /path/to/mydb

# List assets in a datasource
vd3 datasource assets my-datasource -p /path/to/mydb

# Show media availability
vd3 media status -p /path/to/mydb
```

Every command reads `vd3 <noun> <verb> [args]`. The nouns are `db`,
`datasource`, `workset`, `asset`, `layer`, `media`, `eval`, and `ontology`.
Containers (`datasource`, `workset`) own ingest verbs and contained-asset
listings; the `asset` noun is reserved for operations on a known asset.

## Core Concepts


- An **Asset** is a single **Video** or **Imageset** (directory of images).
  - A **Video** implies a temporal relationship and is backed either by:
    - A compressed `.mp4` file.
    - An uncompressed `.tar` of images (a "tar-of-images").
  - An **Imageset:** can be a directory of images (version controlled at the directory) or similarly an uncompressed `.tar` of images.

![Asset Core Type](https://cdn.jsdelivr.net/gh/filitchp/vd3-diagrams@46220e8d94f4a24e3a760bab0d8b545186319cef/assets.svg)
- `vd3` provides consistent iteration of all Asset **frames** straight from the video, directory, or archive.


  Benefits of each **asset** storage mechanism:

  | Video as `.mp4`                               | Video or Imageset as `.tar` ("tar-of-images")                              |
  | --------------------------------------------- | -------------------------------------------------------------------------- |
  | High temporal compression for space savings   | No additional re-encoding (vs image to mp4) and easy to inspect each frame |
  | Stores original data without modification     | Kept intact as one DVC blob - good for version control scaling             |

  NOTE: It is still possible to store an **imageset** as a directory of images, but this is not recommended due to the version control overhead (this leads to long transfer times).

- **Datasource:** groups assets by origin (e.g. `dashcam`, `security-cam`). **Datasource** is all about traceability to the raw source. If you have questions about “where did this come from” you look at the datasource It is required when importing.

- **Workset:** a named subset of assets, optionally organized into packages (folders). Independent of storage layout and used as a general purpose organizational tool for your workspace.

![Assets and Worksets](https://cdn.jsdelivr.net/gh/filitchp/vd3-diagrams@46220e8d94f4a24e3a760bab0d8b545186319cef/worksets.svg)

- **Annotation Layer:** Often referred to as just a **layer**. Layers hold frame-aligned metadata for an asset. By convention they are typically prefixed with a key to describe the type of data (e.g. `gt` for ground truth, `det/yolo-v8` to describe CNN detections). Layers also have a **source**:
  - `GROUND_TRUTH` - the authoritative layer used for computing metrics. Can be derived from human annotations or machine annotations (or a fused combination).
  - `HUMAN_ANNOTATED` - a layer created by humans. May be reviewed or unreviewed. May contain sparsely annotated frames (every N frames).
  - `RESULT` - machine annotations from experiments.

![Annotation Layers](https://cdn.jsdelivr.net/gh/filitchp/vd3-diagrams@46220e8d94f4a24e3a760bab0d8b545186319cef/annotation-layers.svg)

  An asset may have at most one `GROUND_TRUTH` layer **per layer-type family** (its bbox layers — detections/tracks — and its classification layers are counted separately, so a ground-truth bbox layer and a ground-truth classification layer can coexist). Layers can be **git-backed** or **DVC-backed** (recommended at scale). DVC allows fetching layers on demand like media.
  A layer can also be kept purely **local** (uncommitted — for experiments) via `--local`, then promoted to version-controlled layer. See **[Local (uncommitted) layers](#local-uncommitted-layers)**.

- **Imageset Metadata layer:** This is special **layer** for storing arbitrary non-bbox metadata attached per image (labels, attributes, etc). It keeps bulky per-image metadata out of the git-tracked `db/metadata/<datasource>/<asset_name>/imageset.json` manifest. Use `vd3 layer migrate-imageset-extra` to move existing inline content from the `extra` field into a **imageset metadata layer** .

- **Layer Group:** a per-layer identifier that groups the outputs of one *computer-vision solution* together (a CNN detector + a background-model MTI detector + the tracker built from either, all belonging to the same pipeline release). Downstream selectors can organize this by these logical groups to better track experiments. See **[Layer groups](#layer-groups)**.

- **Classification layer + Ontology:** a `classification` **layer** records *fine-grained* classes for a **tracks** layer — its body is an `entries` map keyed by `track_id`, one verdict per track from a single model (`{ontology_node_id, raw_label?, confidence?, chips?, …}`). It names the tracks layer it describes via `target_layer`, and its classes are governed by an **ontology**: a git-versioned, hierarchical taxonomy (e.g. `Vehicle → Truck → Ford F150`) stored in the database, managed with the `vd3 ontology` noun. Run several classification layers over one tracks layer (one per model) to keep every model's verdict side by side for consensus. Import with `vd3 layer add-classification`; inspect with `vd3 layer classifications`.


## Adding Assets

### Videos

```bash
# Single file
vd3 datasource add-video dashcam clip.mp4

# Glob (recursive)
vd3 datasource add-video dashcam 'rawdata/**/*.mp4'

# Tar-of-images video (frames served from the archive; --fps stamps a nominal rate)
vd3 datasource add-video dashcam frames.tar --fps 30

# Force re-import of a duplicate (matched by SHA-256)
vd3 datasource add-video dashcam clip.mp4 --force

# Add and assign to a workset/package
vd3 datasource add-video dashcam clip.mp4 -w my-workset -k batch1
```

### Imagesets

```bash
# Directory of images
vd3 datasource add-imageset my-datasource /path/to/images

# Tar archive
vd3 datasource add-imageset my-datasource images.tar
```
See also [asset and data source name validation rules](#Asset-and-datasource-names).

### Annotation layers

Import VD3 JSON detections/tracks into an existing asset. The second positional is a
layer-name prefix prepended to every `layer` field in the file (e.g. importing a
file with `det/yolo` under `run-1` produces `run-1/det/yolo`):

```bash
vd3 layer add-vd3 clip run-1 results.json -p /path/to/mydb
```

Layers can also be added directly from Python — pass the frame-indexed annotations
plus a `source` (`GROUND_TRUTH` / `HUMAN_ANNOTATED` / `RESULT`) and a `layer_type`
(`detections` / `tracks`):

```python
from vd3storage import VD3Storage

with VD3Storage("/path/to/mydb") as db:
    asset = db.get_asset("dashcam", "clip1")
    db.add_annotation_layer(
        asset.asset_id,
        layer="det/cnn-v1",
        display_name="CNN detector v1",
        annotations={0: [{"class_name": "car", "bbox": [10, 20, 30, 40], "confidence": 0.91}]},
        source="RESULT",
        layer_type="detections",
    )
```

### Layer groups

A `layer_group` labels each annotation layer with the **computer-vision solution** it
belongs to, so multiple outputs from the same pipeline surface together in downstream
tools. This matters as soon as a single "solution" writes more than one layer per asset —
which is the common case for modern CV pipelines. Consider a solution named `sb-v3` that
runs three stages:

| `layer`                      | `layer_type` | `layer_group` | What it holds                                       |
| ---------------------------- | ------------ | ------------- | --------------------------------------------------- |
| `det/sb-v3-cnn`              | detections   | `sb-v3`       | CNN detector output (per-frame boxes)               |
| `det/sb-v3-mti`              | detections   | `sb-v3`       | MTI detector output (from a background model)       |
| `track/sb-v3`                | tracks       | `sb-v3`       | Tracker output that fused the two detection sources |

**Terminology**
- `det` - Detection (e.g. bounding box)
- `track` - A detection with temporal information (e.g. bounding box with track id)
- `MTI` - Moving Target Indicator (e.g. bounding box from background subtraction method)


All three carry `layer_group="sb-v3"`. When you open the asset in VisData, the layer picker
shows **one entry** for `sb-v3`, and clicking it exposes the three underlying overlays
individually — you can toggle the CNN detections, the MTI detections, and the resulting
tracks against each other without hunting through the full flat list.

**Default rule** — when `layer_group` is not set explicitly, vd3 uses the trailing
segment of the `layer` field after the last `/`, so a common naming convention already
groups related layers together for free:

```python
# The det/track pair land in group "sb-v3" via the default rule; the MTI layer
# defaults to its own group -- see below.
db.add_annotation_layer(asset.asset_id, "det/sb-v3",   "SB v3 CNN", cnn_dets, source="RESULT", layer_type="detections")
db.add_annotation_layer(asset.asset_id, "det/sb-v3-mti", "SB v3 MTI", mti_dets, source="RESULT", layer_type="detections")  # default -> "sb-v3-mti"
db.add_annotation_layer(asset.asset_id, "track/sb-v3", "SB v3 tracks", tracks, source="RESULT", layer_type="tracks")
```

Notice the MTI layer defaults to its own group because the trailing segment is
`sb-v3-mti`. To pull it into the shared `sb-v3` group, pass `layer_group=` explicitly:

```python
db.add_annotation_layer(
    asset.asset_id,
    layer="det/sb-v3-cnn",
    display_name="SB v3 CNN",
    annotations=cnn_dets,
    source="RESULT",
    layer_type="detections",
    layer_group="sb-v3",          # <- explicit
)
db.add_annotation_layer(
    asset.asset_id,
    layer="det/sb-v3-mti",
    display_name="SB v3 MTI",
    annotations=mti_dets,
    source="RESULT",
    layer_type="detections",
    layer_group="sb-v3",          # <- share the group with the CNN layer
)
db.add_annotation_layer(
    asset.asset_id,
    layer="track/sb-v3",
    display_name="SB v3 tracks",
    annotations=tracker_output,
    source="RESULT",
    layer_type="tracks",
    layer_group="sb-v3",          # <- and with the tracker built from them
)
```

**When you write a layer JSON directly** — for example when generating results from a
model-serving pipeline that emits VD3 JSON files — set the `layer_group` field in the
body itself:

```json
{
  "layer": "det/sb-v3-mti",
  "display_name": "SB v3 MTI detections",
  "layer_type": "detections",
  "layer_group": "sb-v3",
  "source": "RESULT",
  "frames": { "0": [ ... ], "1": [ ... ] }
}
```

Import it the same way as any VD3 JSON layer (`vd3 layer add-vd3 …`); the `layer_group`
value round-trips into the `asset_layers` table on write and is re-read from the JSON
body on future scans. `layer_group` is stored and queryable, but no CLI listing groups by
it today — the database-wide inventory aggregates by layer *key*:

```bash
vd3 layer list          # per-layer-key coverage: Layer / Types / Assets / %
vd3 asset layers clip1  # per-asset detail
```

### Import COCO Annotations

Import COCO annotations into an existing imageset:

```bash
vd3 layer add-coco my-imageset gt annotations.json \
    --source GROUND_TRUTH --reviewed-all
```

Import a full COCO dataset (creates the imageset and imports annotations in
one step):

```bash
vd3 datasource add-imageset-from-coco my-datasource gt annotations.json \
    --image-root /path/to/images
```

### Git-backed vs. DVC-backed annotations

Annotation layers are **git-backed by default** — the JSON is committed directly, so
it's always present after a clone (ideal for small ground-truth files). Large or bulky
layers can instead be **DVC-backed**, like media: git tracks only a small `.json.dvc`
pointer, and the body is fetched on demand and travels with its media on push/pull.
(Media/video files are always DVC-backed; only annotations offer the choice.)

Add a new layer straight to DVC with `--dvc`:

```bash
vd3 layer add-vd3 clip run-1 results.json --dvc -p /path/to/mydb
```

Already have layers on disk? Use `add-to-dvc` to add them to version control via DVC.
It handles both cases uniformly: a brand-new, not-yet-committed layer is added straight
to DVC, and a layer already tracked by Git is untracked from Git first, then added.
Preview first, then scope by asset/layer or add everything (optionally only layers above
a size):

```bash
vd3 layer add-to-dvc --all --dry-run        # preview what would be added
vd3 layer add-to-dvc --all                  # add every layer
vd3 layer add-to-dvc clip --layer det/qm    # or scope to one asset/layer
vd3 layer add-to-dvc --all --min-size 1     # only layers ≥ 1 MB
```

Use `--layer` to add just one layer while leaving the rest un-added — handy when some
layers on an asset are ready to version but others are still work-in-progress and should
stay uncommitted:

```bash
# Add only det/qm to DVC across every asset; det/wip and any others are left as-is.
vd3 layer add-to-dvc --all --layer det/qm
```

`add-to-dvc` `git rm --cached`s each already-tracked body, `dvc add`s it, and writes the
`.dvc` pointer plus an `annotations/.gitignore` entry. Afterward, commit the new pointers
and `.gitignore`, then push the bodies to your remote:

```bash
vd3 media push --all
```

`add-to-dvc` scans each registered asset's `db/metadata/.../annotations/` directory and acts
on every `*.json` body it finds (subject to `--layer` / `--min-size`) — including a file
dropped in by hand, which is `dvc add`-ed as-is, with no schema check. Importing instead
(`vd3 layer add-vd3 … --dvc`, or import then add) validates the body against the VD3 layer
schema before it is versioned.

### Local (uncommitted) layers

Running experiments you're not ready to share? Add a layer with `--local` and **nothing about it
enters version control** — not the annotations, not even the layer's name. Its JSON body is
excluded through `.git/info/exclude` (which is per-clone and never committed, just like the layer
itself) and its index row goes into a git-ignored `asset_layers_local.csv` sidecar instead of the
committed `asset_layers.csv`. So `git add -A` sweeps up nothing, and your teammates never see rows
pointing at layers they don't have.

```bash
# Import an experiment as a local, uncommitted layer
vd3 layer add-vd3 clip exp-42 results.json --local -p /path/to/mydb

# (COCO too — create the imageset first, then import the layer locally)
vd3 datasource add-imageset-from-coco … then: vd3 layer add-coco porch gt anns.json --local
```

Local layers still behave normally in *your* workspace — they show up in `vd3 asset layers`
(marked `local` in the Storage column) and are readable/queryable. `--local` is mutually
exclusive with `--dvc` (DVC implies sharing).

When an experiment proves out, promote it into version control — a **one-way** operation:

```bash
vd3 layer track-local clip exp-42/det/yolo --dry-run   # preview
vd3 layer track-local clip exp-42/det/yolo             # strip the local flag, git-stage the body,
                                                       # move the row into asset_layers.csv
```

After `track-local` the layer is a normal git-backed layer; commit it to share, or run
`vd3 layer add-to-dvc` if it's large. There is no reverse command (version-controlled → local).

The layer JSON files under `db/metadata/.../annotations/` are the source of truth; the
`asset_layers.csv` / `asset_layers_local.csv` tables are derived caches. If they ever drift
(out-of-band file edits, a `git checkout`, a pull), rebuild them from disk:

```bash
vd3 layer reassociate --dry-run   # preview what would change
vd3 layer reassociate             # rescan every asset, re-split local vs. shared, drop dangling rows
```

`reassociate` also repairs local layers' git exclusions: it re-asserts a `.git/info/exclude` rule
for every local layer on disk — needed after a `git init` (or re-clone) that post-dates them, since
that file is per-clone — and migrates any 1.1.0-era rule out of the tracked `annotations/.gitignore`.

#### Practical usage example

A full round-trip — add an experiment locally, rename it, confirm the annotations never leak
into git, then promote it once it proves out (assumes an asset `clip` and `VD3_DB` set, or pass
`-p <db>`):

```bash
# 1) Add an experiment as a local (uncommitted) layer
vd3 layer add-vd3 clip exp1 results.json --local   # imports under the "exp1/…" prefix, kept local

# 2) It's local — the Storage column shows "local", and the body is excluded from git
vd3 asset layers clip
git status --short                                  # nothing referencing the layer shows up
git check-ignore -v db/metadata/videos/*/clip/annotations/exp1--det--yolo.json
                                                    # -> .git/info/exclude

# 3) Rename it (rename previews by default; --no-dry-run applies)
vd3 layer rename exp1/det/yolo exp2/det/yolo --asset clip --no-dry-run

# 4) Still local after the rename
vd3 asset layers clip                               # now exp2/det/yolo, Storage = local
git add -A && git status --short                    # the renamed body is NOT staged — the
                                                    # exclude rule moved with it

# 5) When you're ready to share it (one-way)
vd3 layer track-local clip exp2/det/yolo            # promotes it into version control
```

`add-vd3`'s second argument is a prefix prepended to each `layer` field in the file, so a file
containing `det/yolo` becomes `exp1/det/yolo` — hence the full key in the rename.

### Layer versioning (status, push, pull, revert)

DVC-backed annotation layers have the same push/pull lifecycle as media, plus tooling to
inspect sync state and undo local edits. These commands are grouped under a **Versioning**
panel in `vd3 layer --help`. All of `push` / `pull` / `revert` scope with
`--asset` / `--datasource` / `--workset` / `--all`.

**See what's synced and what changed locally** — a per-layer counterpart to `vd3 media status`:

```bash
vd3 layer status                     # summary + per-datasource breakdown
vd3 layer status --list              # one row per layer (Datasource/Asset/Layer/Backing/Status)
vd3 layer status --datasource dashcam
```

Each layer reports one of `synced` (on the remote), `local (not pushed)`, `remote-only`
(pointer present, body not pulled), `modified` (edited since its last `dvc add`, or
uncommitted git changes for a git-backed layer), `local` (DVC-backed but no remote
configured), or `git-tracked`.

**Push / pull layer bodies** — a layer-only counterpart to `vd3 media push` / `pull` (which
bundle media *and* layers). Use these to move just the annotations:

```bash
vd3 layer push --datasource dashcam  # upload DVC-backed layers in the datasource
vd3 layer pull --workset my-experiment
vd3 layer push --all                 # every DVC-backed layer in the database
```

> A layer shown as `modified` won't upload its new bytes with a plain push — `dvc push`
> sends what the `.dvc` pointer currently references. Re-record it first
> (`dvc add db/metadata/.../annotations/<layer>.json`), then `vd3 layer push`.

**Revert local modifications** — discard edits and restore the last-recorded content
(DVC-backed bodies from the DVC cache, git-backed *tracked* bodies from git). Dry-run by
default, so you always preview first:

```bash
vd3 layer revert --all               # DRY RUN — lists what would be reverted, changes nothing
vd3 layer revert --all --no-dry-run  # actually restore
vd3 layer revert --asset clip        # scope to one asset
```

Untracked git-backed bodies have no previously-recorded version, so `revert` leaves them
untouched.

## Worksets

```bash
# Create
vd3 workset create "My Experiment"

# Add assets by name or ID
vd3 workset add my-experiment clip-001 clip-002

# ...or by media-path glob (run from the database root; files must be on disk)
cd /path/to/mydb
vd3 workset add my-experiment 'db/media/videos/fc/*.mp4'

# Inspect
vd3 workset list
vd3 workset show my-experiment      # metadata + packages (use `workset layers` for layers)
vd3 workset assets my-experiment    # assets in the workset

# Edit the workset's own attributes
vd3 workset set-description my-experiment "Held-out clips for the v3 eval"
vd3 workset rename my-experiment "My Experiment V2"
vd3 workset rename my-experiment "My Experiment V2" --slug my-experiment-v2

# Remove an asset / delete the workset
vd3 workset remove my-experiment clip-001
vd3 workset delete my-experiment
```

Changing the slug is safe: membership is keyed by the workset's ID and no on-disk path contains
the slug, so the workset's assets come along unchanged. The new slug must be unused.

## Datasources

A datasource is a name carried on its assets, plus an optional description row that exists only to
hold that description:

```bash
vd3 datasource list
vd3 datasource set-description dashcam "Forward-facing 2024 dashcam clips"
```

**Removing one** is therefore ambiguous — forget the description, or delete everything under the
name? — so the command makes you say which:

```bash
vd3 datasource remove dashcam                       # DRY RUN; blocks if the datasource has assets
vd3 datasource remove dashcam --assets              # DRY RUN — lists what would go, changes nothing
vd3 datasource remove dashcam --assets --no-dry-run # delete the assets and the description row
vd3 datasource remove dashcam --assets --media --no-dry-run   # ...and the media files
```

Three things guard it, the same three that guard `vd3 layer remove`:

- **A populated datasource is refused** unless you pass `--assets`, with an error naming the asset
  count. An empty one just drops its description row. `--media` means nothing on its own and is
  rejected without `--assets`.
- **Dry run is the default.** Nothing is deleted until you pass `--no-dry-run`; there is no
  confirmation prompt, so every invocation stays scriptable.
- **The blast radius is printed before anything is deleted**, on both paths — the assets, the
  layer-index rows the cascade drops, the media size (or `not pulled, size unknown` for DVC-backed
  media that isn't local, or `missing from disk` when there is no pointer either), and whether a
  description row goes. The preview and the real run share one rendering, so what you read is what
  happens; the real run then confirms with a `Removed datasource …` line once the work is done.

**The cascade is all-or-nothing.** Files are moved aside rather than destroyed and the affected
rows are snapshotted first, so a failure on the tenth asset — or a failed rename — rolls the whole
operation back and reports `Rolled back — nothing was deleted.` rather than leaving you with half a
datasource. Only once every row and file is dealt with are the staged copies destroyed. In the rare
case where the rollback itself cannot put a file back, vd3 says so — `RECOVERY NEEDED`, with the
paths and the journal that describes them — instead of claiming a clean rollback.

**A killed process is recovered on the next open.** Before anything moves, vd3 snapshots the table
files the removal will rewrite and journals every path it may move — both fsynced — so an
interrupted removal can be finished deterministically: if the deletion had not been committed, rows
*and* media are restored together (a crash midway through writing the tables cannot leave orphan
tags or memberships behind); if it had, the staged copies are binned. Recovery runs before the
database's tables are read, validates every journal entry before moving anything, and leaves a
journal that is unreadable, altered, or names a path outside the database in place for you to look
at rather than guessing — including a journal whose entries do not account for everything sitting
in the staging area. It will also never overwrite a file that has reappeared at an original path:
both copies are kept and reported.

**And it never claims more than it knows.** `Rolled back — nothing was deleted.` appears only after
a rollback verified complete. If files could not be put back you get `RECOVERY NEEDED` with their
locations; if the rows were deleted but recording the outcome failed you get `OUTCOME UNCERTAIN`,
because at that point the journal on disk — not the command — decides what the next open does.

**A delete only ever touches paths inside the database.** An asset's `media_path` is data — it can
be hand-edited in `assets.csv` — so it is validated on write and on load (must be relative, no
`..`), and re-checked at use time against the datasource directory it must live in. A media file
or directory that turns out to be a symlink pointing out of the database is refused, not followed.

Deleting media is not recoverable from the content database itself. Empty per-datasource
directories left under `db/media/` and `db/metadata/` are pruned with `rmdir` only — a stray file
blocks the removal rather than being destroyed.

## Remote Storage

Media files are tracked by DVC. A content database has a single configured remote.

```bash
# Set the remote (replaces any existing one)
vd3 media remote set gs://my-bucket/vd3-data
vd3 media remote show

# Sync (push and pull both accept --workset/-w, --asset/-a, --datasource/-d, --all)
vd3 media push --all
vd3 media pull --workset my-experiment
vd3 media status
```

Supported backends:

| Backend | URL form | Notes |
|---|---|---|
| Google Cloud Storage | `gs://bucket/path` | `gcloud auth application-default login` |
| Amazon S3 | `s3://bucket/path` | Standard AWS credential chain |
| Azure Blob Storage | `azure://container/path` | |
| Google Drive | `gdrive://folder-id` | via `dvc-gdrive` |
| Local / NAS | `/mnt/nas/vd3-backup` | |

## Override DB Path

All commands that accept `--path` also honor the `VD3_DB` environment variable, so
you can point at a database once and drop the `-p` flag from individual commands.
On startup `vd3` automatically loads a `.env` file from the current directory or
the nearest ancestor, so a project-level `.env` containing:

```env
VD3_DB=/path/to/mydb
```

is enough — no `export` or shell sourcing required:

```bash
vd3 asset list
vd3 media status
```

Precedence: explicit `--path` > shell-exported `VD3_DB` > `.env` `VD3_DB` > current
directory. Shell-exported values win over `.env` so you can do one-off overrides
without editing the file.

`--path` on a specific command still overrides the env var.

## Listing & Inspection

```bash
vd3 asset list                       # all assets (cross-container)
vd3 datasource list                  # all datasources
vd3 datasource assets dashcam        # assets in a datasource
vd3 datasource assets dashcam --paths      # one media path per line
vd3 datasource assets dashcam --filenames  # one filename per line
vd3 datasource layers dashcam        # annotation layers across the datasource
vd3 workset assets my-experiment     # assets in a workset
vd3 workset layers my-experiment     # annotation layers across the workset
vd3 asset layers clip                # annotation layers on an asset
vd3 layer list                       # annotation layers across the whole database (per-layer coverage)
vd3 layer status                     # per-layer DVC sync state + local modifications
vd3 asset show clip                  # asset details
vd3 db info                          # database overview
vd3 db query "SELECT ..."            # raw DuckDB SQL against the CSV tables
vd3 db doctor                        # consistency check across tables, media and metadata
```

## Health Checks

`vd3 db doctor` scans a content database for the ways its CSV tables and its on-disk tree can
disagree, and reports each finding as `OK`, `WARN` or `ERROR`. Without `--fix` the checks only
read, and it exits non-zero if anything `ERROR`-level turns up, so it drops straight into CI.
(One caveat worth knowing: *opening* a database applies any pending schema migration and rewrites
`db/tables/.schema_version` — true of every `vd3` command, `db doctor` included. The `schema`
check reads the marker before that open, so it reports what was on disk rather than what the open
left behind.)

```bash
vd3 db doctor          # report only (exit 1 on any ERROR)
vd3 db doctor --fix    # apply the safe repairs, then list what still needs a human
```

What it checks:

| Check | Looks for |
|---|---|
| `schema` | `.schema_version` present and matching the version this vd3 understands |
| `layer-tables` | `asset_layers.csv` / `asset_layers_local.csv` vs. the annotation JSONs on disk |
| `ground-truth` | Assets carrying more than one `GROUND_TRUTH` layer |
| `media` | Assets whose `media_path` has neither a local file nor a `.dvc` pointer |
| `media-pointers` | `.dvc` pointers under `db/media/` with no asset row behind them |
| `media-pointers-unpulled` | Orphaned `.dvc` pointers whose media was never fully pulled (never auto-removed) |
| `workset-assets` | Membership rows referencing a missing asset or workset |
| `tags` / `evaluations` | Rows referencing a missing asset / workset scope |
| `metadata-dirs` | Directories under `db/metadata/` with no row in `assets.csv` |
| `asset-metadata` | Imported assets whose metadata directory is gone |

`--fix` applies only the provably safe repairs: rebuilding the layer index, deleting orphaned
`.dvc` pointer files **whose media is still here locally**, and dropping stale workset-membership
rows. **No asset, media file or annotation body is ever deleted** — duplicate ground truth,
missing media and a schema version from the future are reported for a human to resolve. An
orphaned pointer whose media was never pulled is reported too, never deleted: with no local copy
that pointer is the only record of the content's md5, and removing it would strand the bytes in
the DVC cache/remote. The doctor is entirely offline (it never
shells out to DVC); for remote sync state use `vd3 media status`.

## Exporting

```bash
# Extract frames from a video or images from an imageset
vd3 asset export-frames clip -o ./out
```

## Library API

The CLI is a thin wrapper around `VD3Storage`, which is also usable directly.

```python
from vd3storage import VD3Storage, Asset, Workset  # Tag, WorksetAsset also exported

# Open an existing database (or use VD3Storage.init(path) to create one)
storage = VD3Storage("/path/to/mydb")

# Browse assets
for a in storage.list_assets(datasource="dashcam"):
    print(f"{a.name} ({a.asset_type}): {a.frame_count} frames @ {a.nominal_fps} fps")

# Look up by (datasource, name) or by ID
clip = storage.get_asset("dashcam", "clip-001")
clip = storage.get_asset_by_id("3f1a...")

# Import a video
asset = storage.import_video("clip.mp4", datasource="dashcam")

# Resolve where the media file lives on disk
storage.resolve_media_path(clip)

# Annotation layers
storage.list_annotation_layers(clip.asset_id)
storage.read_annotation_layer(clip.asset_id, "gt")

# Worksets
ws = storage.create_workset("My Experiment")
storage.add_asset_to_workset(ws.workset_id, clip.asset_id, package="batch1")
storage.list_workset_assets(ws.workset_id)

# Raw DuckDB SQL against the underlying CSV tables
rows = storage.execute_sql("SELECT name, frame_count FROM assets WHERE asset_type = 'video'")
```

Other useful methods: `import_imageset`, `import_coco`, `import_coco_dataset`, `import_result`, `export_coco`, `open_video`, `open_imageset`, `get_frame_image`, `add_tag`, `is_media_available`, `pull`, `push`. Inspect `help(VD3Storage)` for the full surface.

## Versioning & stability

The package follows [Semantic Versioning](https://semver.org). All notable
changes are recorded in [CHANGELOG.md](CHANGELOG.md), and every release is
tagged `vX.Y.Z` in git.

**Since 1.0.0 (2026-07-13) the package follows strict SemVer**: MAJOR = breaking,
MINOR = additive, PATCH = fix. Every breaking change is called out under a
`### Breaking` heading in the CHANGELOG entry for that release.

(Before 1.0.0, minor bumps such as 0.2 → 0.3 could contain breaking changes;
the pre-1.0 entries in the CHANGELOG should be read with that in mind.)

### Public API

A change is "breaking" only if it alters one of the following:

1. **Names re-exported from the top-level `vd3storage` package** (i.e. listed
   in `vd3storage.__all__`):
   - `VD3Storage` and its documented methods
   - `AlreadyInitializedError`
   - The model classes `Asset`, `Tag`, `Workset`, `WorksetAsset`
     (including their field names and types)
   - `__version__`
2. **The `vd3` CLI** — command names, option names, exit codes, and the
   documented input file formats (VD3 JSON, COCO).
3. **The on-disk layout of a content database** — directory structure under
   `db/`, CSV table schemas (tracked by `SCHEMA_VERSION` in `db/tables/`), the
   shape of `video.json` / `imageset.json` / annotation JSON files, and the
   structure of generated `pyproject.toml` / `.gitignore`.

Everything else is **internal** and may change without a major-version bump
even if it is reachable via an import path. That includes the `vd3storage.orm`,
`vd3storage.dvc`, `vd3storage.media`, `vd3storage.importers`,
`vd3storage.exporters`, and `vd3storage.cli` submodules; helper functions in
`vd3storage.storage` that start with `_`; and the on-disk format of files
written into `.dvc/` (those belong to DVC).

### Deprecations

When a public API needs to change incompatibly, the old form keeps working and
emits `DeprecationWarning` for at least one minor release before being removed.
Current deprecations are listed in the CHANGELOG under each release's
`### Deprecated` heading.

To surface them in your own code:

```bash
python -W "default::DeprecationWarning:vd3storage" your_script.py
```

## CLI Reference

```
vd3 --help                       Top-level help
vd3 <noun> --help                Help for a noun
vd3 <noun> <verb> --help         Help for a specific command
```

Every command reads `vd3 <noun> <verb> [args]`. Positionals carry identity
(target → composite parts → payload); flags carry modifiers
(`--paths`, `--filenames`, `--source GROUND_TRUTH`, ...).

| Command | Description |
|---|---|
| `db init` | Initialize a content database (defaults to cwd) |
| `db info` | Show database overview |
| `db query` | Run raw DuckDB SQL against the CSV tables |
| `db doctor` | Check the database for consistency problems (`--fix` applies the safe repairs) |
| `datasource list` | List datasources |
| `datasource show` | Show datasource stats |
| `datasource assets` | List assets in a datasource |
| `datasource layers` | List annotation layers across a datasource (per-layer coverage) |
| `datasource add-video` | Import video files into a datasource |
| `datasource add-imageset` | Import an imageset (directory or tar) into a datasource |
| `datasource add-imageset-from-coco` | Import a COCO dataset as imageset + layer |
| `datasource set-description` | Set a datasource's human-readable description |
| `datasource remove` | Remove a datasource: its description row, and with `--assets` (required if it has any) every asset in it, plus `--media` for the files; dry-run by default (`--no-dry-run` to apply) |
| `workset create` | Create a workset |
| `workset list` | List worksets |
| `workset show` | Show workset metadata + packages |
| `workset assets` | List assets in a workset |
| `workset layers` | List annotation layers across a workset (per-layer coverage) |
| `workset add` | Add assets to a workset |
| `workset remove` | Remove an asset from a workset |
| `workset set-description` | Set a workset's description |
| `workset rename` | Rename a workset (`--slug` also changes its slug; membership is unaffected) |
| `workset delete` | Delete a workset (assets are kept) |
| `asset list` | List every asset (cross-container) |
| `asset layers` | List annotation layers on an asset |
| `asset show` | Show asset details |
| `asset remove` | Delete an asset |
| `asset export-frames` | Extract frames from a video or imageset |
| `asset split` | Set dataset split affinity (`trainval`/`train`/`val`/`test`/`omit`, or `""` for unassigned) |
| `asset star` / `unstar` | Set or clear the queryable bookmark flag (`asset list --starred` filters on it) |
| `asset set-comments` | Set the free-text comments field on an asset |
| `asset extra` / `extra-set` | Read or write a consumer's namespaced blob in `extra_json` (sibling namespaces preserved) |
| `asset environment` | Set capture environment (`urban`, `rural`, `highway`, `marine`, …) |
| `asset placement` | Set sensor placement (`outdoor`, `indoor`, `various`, `unknown`) |
| `asset time-of-day` | Set lighting conditions (`day`, `night`, `dawn_dusk_cloudy`, `unknown`) |
| `asset spectrum` | Set sensor spectrum (`visible`, `infrared`, `various`, `unknown`) |
| `asset synthetic` | Mark whether the media was synthetically generated (`no`, `yes`, `unknown`) |
| `layer list` | List annotation layers across the whole database (per-layer coverage) |
| `layer status` | Show per-layer DVC sync state (synced / not pushed / remote-only) and local modifications (`--asset`/`--datasource`/`--workset`, `--list`) |
| `layer push` | Push DVC-backed annotation layers to the remote by scope (`--asset`/`--datasource`/`--workset`/`--all`) |
| `layer pull` | Fetch DVC-backed annotation-layer bodies from the remote by scope |
| `layer revert` | Discard local modifications to layers (DVC cache / git); dry-run by default (`--no-dry-run` to apply) |
| `layer add-coco` | Import COCO annotations into an existing imageset (`--local` keeps it uncommitted) |
| `layer add-vd3` | Import VD3 JSON detections/tracks under a layer-name prefix (`--dvc` stores them DVC-backed; `--local` keeps them uncommitted) |
| `layer add-classification` | Import a fine-grained classification file as a `classification` layer (`--target-layer`, `--ontology`, `--source`) |
| `layer classifications` | Print a classification layer's per-track verdicts with node paths resolved (`--track`, `--node`) |
| `layer rename` | Rename a layer key on one asset (`--asset`) or every asset (`--all`); dry-run by default (`--no-dry-run` to apply) |
| `layer remove` | Delete a layer from one asset (`--asset`) or every asset that has it (`--all`, required); dry-run by default (`--no-dry-run` to apply) |
| `layer add-to-dvc` | Add layers to version control via DVC, or migrate Git-backed layers to DVC (`--layer`, `--all`, `--min-size`, `--dry-run`) |
| `layer track-local` | Promote a local (uncommitted) layer into version control — one-way (`--dry-run` to preview) |
| `layer reassociate` | Rebuild `asset_layers.csv` / `asset_layers_local.csv` from the on-disk layer JSONs; drops rows for deleted assets (`--dry-run` to preview) |
| `layer source` | Re-classify a layer's source (`GROUND_TRUTH` / `HUMAN_ANNOTATED` / `RESULT`) |
| `layer promote` | Promote a result layer to ground truth |
| `layer approve` / `reject` | Record a human review verdict on a ground-truth layer (`--by`, `--reason`); surfaces as `Asset.gt_reviewed` |
| `layer unreview` | Clear the review verdict and its audit fields |
| `layer pin` / `unpin` | Set or clear the asset's canonical `RESULT` layer (`Asset.pinned_result`) |
| `layer migrate-source` | Rewrite legacy `human`/`machine` source values to the current taxonomy |
| `layer migrate-imageset-extra` | Move inline `imageset.json` `extra` data into an `image_metadata` layer |
| `media status` | Show media availability |
| `media push` | Push media to remote storage |
| `media pull` | Pull media from remote storage |
| `media remote set` | Set the remote storage URL |
| `media remote show` | Show the configured remote |
| `eval add` | Record an evaluation/metrics blob for an asset or workset (`--run`, `--metrics`, `--config`) |
| `eval list` | List recorded evaluations |
| `eval show` | Show one evaluation's metrics and config |
| `eval update` | Correct a stored evaluation's `--metrics` / `--config` / `--run` (scope and id are immutable) |
| `eval remove` | Delete one evaluation by id; dry-run by default (`--no-dry-run` to apply) |
| `ontology create` | Create a new, empty ontology (a hierarchical class taxonomy) |
| `ontology list` | List ontologies (slug, name, node count, roots) |
| `ontology show` | Show an ontology as a tree (`--flat` for a path table; `--node <ref>` scopes to a subtree) |
| `ontology add-node` / `rename-node` / `move-node` / `remove-node` | Edit the tree; a node ref is a `node_id` or a path (`--ontology` when the db has more than one) |
| `ontology import` / `export` | Bulk-create a tree from nested JSON / dump it back |
| `ontology rebuild-paths` | Recompute every node's `path` from `parent_id` (mirrors `layer reassociate`) |


## Misc
### Asset and datasource names

An asset's name and its datasource are both used verbatim as URL path components and on-disk
metadata/media path segments (`…/<datasource>/<name>`), so both are validated to stay safe in those
roles.

**Asset name** - the `--name` you pass to an ingest command, or the source filename it defaults to

- **Allowed characters:** letters, digits, and `-` `_` `.` `/` — all URL-safe.
- **Slashes are path separators.** `--name batch1/clip3` nests the asset one directory deeper.
  Segments between slashes must be non-empty (no leading/trailing `/`, no `//`) and must not be `.`
  or `..`.
- **Length:** up to 200 characters.

**Datasource** — same rules, but stricter: because a datasource is a single path/URL segment (never
nested), it may contain **no slashes at all** (neither `/` nor `\`). Allowed characters are letters,
digits, and `-` `_` `.`; it may not be `.` or `..`; up to 200 characters.

Invalid names are rejected up front (before any file is copied) with a clear error, and are also
validated when a database is loaded — so a hand-edited `assets.csv` / `datasources.csv` can't smuggle
in an unsafe value.
