Metadata-Version: 2.4
Name: dustclust
Version: 0.1.0
Summary: DustClust: interactive hardware inventory clustering with embeddings and a web visualization
Author: DustClust contributors
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Requires-Dist: python-multipart
Requires-Dist: pandas
Requires-Dist: openpyxl
Requires-Dist: numpy
Requires-Dist: sentence-transformers
Requires-Dist: scikit-learn

# DustClust

**DustClust** (`import DustClust`) is an interactive web application that clusters dirty hardware inventory data using semantic embeddings and visualizes the results in a D3.js radial network graph. The backend runs live clustering with configurable thresholds and matches clusters to real-world devices from the iFixit catalog plus a custom device list.

Install from PyPI as **`dustclust`** (`pip install dustclust`).

## Features

### GUI

- **Interactive Network Graph**: Zoom, pan, and drag nodes. Hover for quick tooltips; click to open the detail panel.
- **Search Clusters**: Real-time search by category, subcategory, or record content (e.g., Router, iPhone, 4331).
- **Clustering Threshold**: Adjust the similarity threshold (0.10–0.90) and click **Recalculate Clusters** to re-run clustering.
- **Min Cluster Size**: Filter out small clusters with a slider.
- **Upload Dataset**: Upload your own `.csv` or `.xlsx` file. The server generates embeddings and clusters on the fly.
- **Reset View**: Reset zoom and clear search/filters.
- **Detail Panel**: Click any node to see:
  - **Category / Subcategory** (e.g., Computer Hardware / Mouse)
  - Sample records
  - **Matched iFixit Device** (when a cluster matches a known device, with link to iFixit)

### Backend

- **Embedding-based clustering**: Uses sentence-transformers with `paraphrase-multilingual-MiniLM-L12-v2` (or another HuggingFace model ID) for semantic similarity.
- **Hebrew & multilingual support**: The model and tokenization support Hebrew and other Unicode scripts. Category inference includes Hebrew keywords (e.g. מסך, מקלדת, עכבר).
- **Dynamic category inference**: Token-based matching against iFixit categories and `custom_devices.py`.
- **Subcategory inference**: Finer-grained labels (e.g., Keyboard, Monitor, Printer) derived from the device list.
- **Device matching**: IDF-scored matching of cluster records to iFixit devices and custom entries.

## Device List

The app uses two device sources:

1. **iFixit catalog** (`ifixit_devices.json`): Run `fetch_ifixit_devices.py` to populate.
2. **Custom devices** (module `DustClust.custom_devices`): Generic hardware that supplements iFixit for category/subcategory inference and device matching.

### Custom Device Categories

| Category | Subcategories / Types |
|---------|------------------------|
| **SIM Card** | Nano, Micro, Standard, eSIM |
| **Cable** | USB-C, Lightning, HDMI, DisplayPort, Ethernet, VGA, DVI, Thunderbolt, SATA, etc. |
| **Adapter** | USB hubs, HDMI/DisplayPort adapters |
| **Storage** | MicroSD, SD, Flash Drive, HDD, SSD, NAS, CD/DVD/Blu-ray drives |
| **PC Component** | RAM, CPU, GPU, Motherboard, PSU, SSD, Cooling, Thermal paste |
| **Computer Hardware** | Keyboard, Mouse, Laptop, Monitor, Webcam, Dashcam, Projector, Printer |
| **Audio** | Headphones, Earbuds, Microphone |
| **Telecom** | Router, Modem, Network Switch, Access Point, enterprise gear |

Edit `src/DustClust/custom_devices.py` (or the installed package’s `custom_devices.py`) to add or adjust devices. Each entry has `name`, `category`, `subcategory`, and optional `url`.

### Hebrew / Multilingual Datasets

The app works with Hebrew and other Unicode datasets. The default model (`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`) supports 50+ languages. Category inference includes Hebrew boost keywords (מסך, מקלדת, עכבר, כבל, etc.), and `custom_devices.py` has Hebrew device entries for matching. Upload a Hebrew CSV and the clustering and categorization will work out of the box.

## Getting Started

### Install with pip

From the repository root (editable install for development):

```bash
pip install -e .
```

Or from a sdist/wheel once published:

```bash
pip install dustclust
```

This installs the **`dustclust`** CLI and the importable package **`DustClust`**.

### Prerequisites

- Python 3.10+
- Dependencies are declared in `pyproject.toml` (installed automatically with `pip install`).

### Optional: conda and iFixit cache

```bash
conda create -n dustclust python=3.11
conda activate dustclust
pip install -e .
```

Fetch / refresh the iFixit device catalog into the current directory (optional; a copy may already be bundled in the package):

```bash
python fetch_ifixit_devices.py
```

### Running the App

1. Start the server (any of these):

```bash
dustclust
# or
python -m DustClust
# or, from a clone without installing
python server.py
```

2. Open [http://localhost:8001](http://localhost:8001) in your browser.

**Command-line options** (run `dustclust --help` for details):

| Option | Default | Description |
|--------|---------|-------------|
| `--model` | `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | Path to local model folder or HuggingFace ID |
| `--device` | `cpu` | Device for embeddings: `cpu` or `cuda` |
| `--threshold` | `0.3` | Default clustering threshold (0.1–0.9) |
| `--batch-size` | `512` | Batch size for embedding generation |
| `--data` | *(bundled sample in the package)* | Path to CSV dataset; omit to use the bundled sample |
| `--host` | `localhost` | Host to bind |
| `--port` | `8001` | Port to run on |
| `--no-reload` | — | Disable auto-reload on file changes |

The server loads the default dataset (`data/dirty_hardware_data_40k.csv`) and **serves the GUI immediately**; initial embeddings run in the background (the graph appears when they finish—see `/api/status`). Use **Recalculate Clusters** or **Upload Dataset** to change the data or clustering.

## Model

The default embedding model is `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`. Use `--model` to pass a local model folder or another HuggingFace model ID.

## Data

CSV files live in the `data/` folder. The default dataset is `data/dirty_hardware_data_40k.csv`. Use `--data` or `--input`/`--output` to point scripts at other paths.

## Data Processing Workflow

- `dataset_generator.py`: Generates raw `data/dirty_hardware_data_40k.csv`.
- `cluster_hardware` / CLI **`cluster-hardware`**: Produces `data/clustered_output.csv` from embeddings and clustering.
- `prepare_viz_data.py`: Builds static `src/DustClust/cluster_viz/data.json` for the legacy static workflow.
- `DustClust.server` / CLI **`dustclust`**: Serves the live app with on-demand clustering and device matching.
