Metadata-Version: 2.4
Name: dqm-ml-images
Version: 2.0.1
Summary: Python library designed provide core dqm-ml metrics without huge dependencies, as well as common API shared by metrics
Author-email: SafenAI <support@safenai.io>
License-Expression: Apache-2.0
Project-URL: Homepage, https://irt-systemx.github.io/dqm-ml
Project-URL: Documentation, https://irt-systemx.github.io/dqm-ml
Project-URL: Repository, https://github.com/IRT-SystemX/dqm-ml
Keywords: ml,metrics,data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: dqm-ml-core
Requires-Dist: pillow>=12.1.1
Requires-Dist: scipy>=1.7.0

# DQM-ML Images

Image feature extraction package for DQM-ML V2. Provides metrics for assessing image dataset quality.

## Installation

```bash
pip install dqm-ml-images
```

> **Note:** `dqm-ml-images` provides **Features Processors** only — no CLI or job orchestration. Use directly via Python or with `dqm-ml-job` for YAML config execution.

## Quick Start: Generate Synthetic Test Images

Create `data/images.parquet` with 5 synthetic 50×50 RGB images — minimalist example:

```python
# generate_data.py
import io
import numpy as np
from pathlib import Path
from PIL import Image
import pyarrow as pa
import pyarrow.parquet as pq

rng = np.random.default_rng(42)
Path("data").mkdir(exist_ok=True)

images = []
for _ in range(5):
    # 50x50 RGB synthetic image
    arr = rng.integers(0, 255, (50, 50, 3), dtype=np.uint8)
    img = Image.fromarray(arr, mode="RGB")
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    images.append(buf.getvalue())

table = pa.table({"image_bytes": images})
pq.write_table(table, "data/images.parquet")
print(f"Generated {len(images)} images -> data/images.parquet")
```

```bash
python generate_data.py
```

## Usage

### Using Python Directly

> **Note:** See [Quick Start](#quick-start-generate-synthetic-test-images) to generate `data/images.parquet` with synthetic test images.

```python
import pandas as pd
from dqm_ml_images import VisualFeaturesProcessor
from dqm_ml_core import ProcessorRunner

# Load synthetic images from parquet (generated by Quick Start script)
df = pd.read_parquet("data/images.parquet")  # columns: image_bytes

# Configure processor (expects "image_bytes" column)
processor = VisualFeaturesProcessor(
    name="image_quality",
    config={
        "columns": {"input": ["image_bytes"]},
        "features": ["luminosity", "contrast", "blur", "entropy"],
        "grayscale": True,
        "normalize": True,
        "laplacian_kernel": "3x3"
    }
)

# Run using ProcessorRunner (high-level API)
runner = ProcessorRunner()
features = runner.run(df, [processor])

print(f"Luminosity: {features['image_bytes_luminosity']}")
print(f"Contrast: {features['image_bytes_contrast']}")
print(f"Blur: {features['image_bytes_blur']}")
print(f"Entropy: {features['image_bytes_entropy']}")
```

### With dqm-ml-job

> **Note:** See [Quick Start](#quick-start-generate-synthetic-test-images) to generate `data/images.parquet` with synthetic test images.

For running from a YAML config, install together with `dqm-ml-job`:

```bash
pip install dqm-ml-job dqm-ml-images
```

Create a YAML config file (e.g., `config.yaml`):

```yaml
dataloaders:
  loaders:
    - name: images
      type: parquet
      path: data/images.parquet
      batch_size: 100

features:
  outputs:
    path: output/features.parquet
  processors:
    - name: image_quality
      type: image_features
      columns:
        input: ["image_bytes"]
      features: [luminosity, contrast, blur, entropy]
      grayscale: true
      normalize: true
      laplacian_kernel: "3x3"
```

Execute from Python:

```python
from dqm_ml_job.cli import execute

# Execute a data quality job from a YAML config
execute(["-p", "config.yaml"])
```

Or from the command line:

```bash
python -m dqm_ml_job.cli -p config.yaml
```

## Features

| Feature | Description |
|---------|-------------|
| **Luminosity** | Mean gray level — measures overall brightness |
| **Contrast** | RMS contrast — measures tonal range |
| **Blur** | Variance of Laplacian — estimates sharpness/focus |
| **Entropy** | Shannon entropy — measures information content |

## Adding a Custom Feature

Features are currently added directly to the `VisualFeaturesProcessor` class. There are five locations to update:

### 1. Define the output column name

Add to `DEFAULT_OUTPUTS` in `visual_features.py`:

```python
DEFAULT_OUTPUTS: dict[str, str] = {
    "luminosity": "luminosity",
    "contrast": "contrast",
    "blur": "blur",
    "entropy": "entropy",
    "colorfulness": "colorfulness",  # new
}
```

### 2. Add to validation tuple

Update the tuple in `_validate_output_features()`:

```python
for k in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):
```

### 3. Add to generated features

Update the tuple in `generated_features()`:

```python
for fk in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):
```

### 4. Write a computation helper

Add a static or instance method:

```python
@staticmethod
def _colorfulness(gray: np.ndarray) -> float:
    """Mean saturation as a simple colorfulness proxy."""
    # gray is already grayscale at this point if grayscale=True;
    # for a real colorfulness metric the method would need RGB input.
    return float(np.mean(gray))
```

### 5. Wire into the dispatch loop

Add a branch in `compute_features()`:

```python
for fk in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):
    func = {"luminosity": np.mean, "contrast": np.std, "colorfulness": np.mean}.get(fk)
    if fk == "blur":
        arr = self._compute_scalar_feature(gray_images, self._variance_of_laplacian, True)
    elif fk == "entropy":
        arr = self._compute_scalar_feature(gray_images, self._entropy, True)
    elif fk == "colorfulness":
        arr = self._compute_scalar_feature(gray_images, self._colorfulness, True)
    else:
        arr = self._compute_scalar_feature(gray_images, func, self.normalize)
    result[self._output_column_name(image_column, fk)] = arr
```

### 6. (Optional) Add to the Pydantic default

If you want the feature on by default, add it to `ImageFeaturesProcessorConfig.features` in `processors.py`:

```python
features: list[str] = Field(
    default=["luminosity", "contrast", "blur", "entropy", "colorfulness"],
)
```

## Output

The processor adds these columns to your data:

- `luminosity`
- `contrast`
- `blur_level`
- `entropy`

## Requirements

- `opencv-python`
- `pillow`
- `numpy`

## Dependencies

DQM-ML is modular. For visual features:

```bash
# Minimal: use as library only
pip install dqm-ml-images

# For YAML config execution
pip install dqm-ml-job dqm-ml-images

# Full stack with all metrics
pip install dqm-ml-job dqm-ml-core dqm-ml-images dqm-ml-pytorch
```

## See Also

- [Visual Features Documentation](https://safenai.github.io/dqm-ml-workspace/docs/metrics/visual_features/)
- [Configuration Guide](https://safenai.github.io/dqm-ml-workspace/docs/configuration/)
