Metadata-Version: 2.4
Name: clearclutter
Version: 0.1.3
Summary: Non-destructive photo/video dedup: scan a drive, review duplicates in a Google-Photos-style UI, move rejects to a bin folder
Author: Manikandan Balasubramanian
License: MIT
Project-URL: Homepage, https://github.com/ManikandanBalasubramanian/ClearClutter
Keywords: photos,dedup,duplicates,phash,photo-management
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Multimedia :: Graphics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pillow
Requires-Dist: pillow-heif
Requires-Dist: imagehash
Requires-Dist: xxhash
Requires-Dist: numpy
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Provides-Extra: faces
Requires-Dist: insightface; extra == "faces"
Requires-Dist: onnxruntime; extra == "faces"
Requires-Dist: opencv-python-headless; extra == "faces"
Dynamic: license-file

# ClearClutter

Non-destructive photo/video dedup for a messy drive, with a Google-Photos-style
review UI. Scans a source tree into a SQLite index, puts **every file into a
group** (duplicates together, unique files as groups of one), and gives you a
daily verify queue. **By default it never deletes anything** — rejected copies
move to `<root>/ClearClutter_bin/` (folder structure preserved) for you to
delete manually, every move is logged, and the last batch is always undoable.
(Permanent deletion exists as an explicit, confirmed opt-in.)

## Install

```bash
pip install clearclutter            # from PyPI, once published
# or from a checkout:
pip install .
# with face recognition ("People"):
pip install "clearclutter[faces]"
```

## Usage

```bash
clearclutter /Volumes/WD            # that's it — review UI at http://127.0.0.1:8177
```

Point it at a drive and everything else happens from the UI: **Sync** scans
and groups (with live progress), and **People → Detect faces** runs face
clustering. The individual commands still exist for scripting:

```bash
clearclutter scan /Volumes/WD       # index the drive (incremental)
clearclutter group /Volumes/WD      # build groups (dupes + singles)
clearclutter faces /Volumes/WD      # optional: cluster faces by person
clearclutter serve /Volumes/WD      # review UI
```

## Background jobs

Sync, apply and face detection run as **background jobs**: a panel in the left
nav shows live progress with pause / resume / cancel, and you keep browsing
and reviewing while they run. The bandwidth selector in the top bar
(Full speed / Balanced / Gentle) caps how fast jobs read the disk, so a scan
never makes the UI unusable on a slow drive.

**The index lives on the drive itself** — `<root>/.clearclutter/` holds the
SQLite database, thumbnails, and the move audit log, so every disk is fully
self-contained and travels with its data. With several disks plugged in, run
one `clearclutter serve <root>` per disk; each instance picks the next free
port automatically and shows its drive name in the top bar.

## Staying in sync

- On server start, a fast background probe (walk + stat, no hashing) counts
  files that are new, changed, or missing versus the index; the **Sync**
  button shows the number.
- **Sync** rescans incrementally (only new/changed files are hashed) and
  rebuilds groups, with live progress. New files land as unverified groups —
  straight into your review queue.
- **Manually moved files are reconciled by content hash**: if you move or
  rename a verified file outside the app, the rescan matches the "missing"
  row to the new path and carries its verified flag and tags over.
- Files that are truly gone (deleted or moved off-drive) show as *broken
  links* in the top bar with a one-click **clean up** (index rows only —
  no files are ever touched).

## How grouping works

| Group type | Signal |
|---|---|
| `exact` | identical bytes (xxh3) — any name, folder, or source |
| `near` | perceptual hash within a tunable threshold — re-encodes, resizes, Google Photos re-downloads, bursts |
| `live` | Apple ContentIdentifier — Live Photo HEIC+JPEG+MOV trios (keeper: the HEIC) |
| `video_time` | videos captured within 120 s — manual review aid |
| `single` | everything else: a unique file is its own group of one |
| `manual` | groups you created by splitting others |

Names are never used for matching, so same-name-different-photo never groups.
The suggested keeper (★) is scored by format (RAW > HEIC > JPEG > PNG),
resolution, EXIF date, GPS, camera, and size — override with one click.
**Degrouping** wrongly-matched photos splits them into their own `manual`
group; never-pair rules and manual groups both survive regrouping.

## The verify workflow

Every file is **unverified** until you deal with its group:

- **Keep ★, bin rest** — queues the duplicates for the bin and marks the whole
  group verified.
- **Verify** — marks a group correct as-is (singles, or groups where you keep
  everything).
- Bulk-select photos in any timeline and hit **Verify** in the action bar.

**Groups (Unverified)** in the left nav is your daily queue; the badge shows
what's left. The **Verified** view shows everything you've already cleared.

## The UI

Left nav: **All Data · Photos · People · Location · Verified · Favourites ·
Groups (Unverified) · Videos**, plus the bin, the pending-actions queue and
the background-jobs panel. Timelines use a justified grid with date headers,
hover-select circles, and a bulk action bar (verify / favourite / tag / move /
bin — one request for the whole selection). Videos **play on hover** in the
timeline and play in-card (muted, with scrubbing) in group review, so
comparing near-identical clips doesn't mean bouncing through a full-screen
player; large videos stream with HTTP range requests, so playback and seeking
are instant. Clicking a photo opens a lightbox with arrow-key navigation and
an info panel. **People** clusters faces (incremental — names survive
re-runs), can be started from the UI, and clusters of the same person can be
**merged**. **Location** clusters GPS on a ~1 km grid. Tags filter any
timeline from a dropdown. Light and dark theme follow the OS.

Applying moves: **Pending actions → Dry run → Apply** — apply runs in the
background with progress. Undo restores the last batch. If you'd rather skip
the bin folder entirely, an explicit opt-in checkbox deletes binned files
permanently (confirmed, and not undoable — the default remains
non-destructive). Tags live in SQLite only — they're a semantic view,
independent of folders.

## Built for volume

Designed around a 1 TB / few-hundred-thousand-file library: one streaming read
per file (hash computed while reading), incremental rescans skip unchanged
files by size+mtime, thumbnails are sharded across 256 subdirectories, group
statistics are precomputed at build time (the review list is a pure indexed
read — no aggregation per page), timelines use an expression index on the
capture date, singleton groups insert in bulk, and every list in the UI is
paginated with infinite scroll. Scans and syncs run in a background thread
with progress — the UI never blocks.

## Notes

- Workspace (index, thumbnails, `moves.jsonl`): `<root>/.clearclutter`
  (`--workspace` to override)
- `exiftool` on PATH enables Live Photo pairing and video dates (`brew install exiftool`)
- phash threshold 6 is conservative; raise toward 10 to catch heavier edits
- NTFS on macOS needs Paragon/Tuxera for the apply step; scanning is read-only
