Metadata-Version: 2.5
Name: mlvbench
Version: 1.0.0
Summary: Human vs. Machine Mid-Level Vision
Project-URL: Homepage, https://github.com/mtangemann/mlvbench
Project-URL: Repository, https://github.com/mtangemann/mlvbench.git
Project-URL: Documentation, https://mtangemann.github.io/mlvbench
Project-URL: Issues, https://github.com/mtangemann/mlvbench/issues
Author-email: Matthias Tangemann <mtangemann@cs.toronto.edu>, Benjamin Lo <benjamin.lo2@mail.mcgill.ca>
License: Copyright 2026 Matthias Tangemann
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.14
Requires-Dist: coloredlogs>=15.0.1
Requires-Dist: datasets>=3.0
Requires-Dist: diffusers[torch]>=0.37.1
Requires-Dist: einops>=0.8.2
Requires-Dist: ipykernel>=7.2.0
Requires-Dist: matplotlib>=3.10.8
Requires-Dist: numpy>=2.4.2
Requires-Dist: opencv-python>=4.13.0.92
Requires-Dist: pillow>=12.1.1
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: quarto>=0.1.0
Requires-Dist: scikit-image>=0.26.0
Requires-Dist: scikit-posthocs>=0.14.0
Requires-Dist: seaborn>=0.13.2
Requires-Dist: submitit>=1.5.4
Requires-Dist: tabulate>=0.9.0
Requires-Dist: timm>=1.0.25
Requires-Dist: torch>=2.10.0
Requires-Dist: torcheval>=0.0.7
Requires-Dist: transformers>=5.5.0
Requires-Dist: typer>=0.24.1
Requires-Dist: tyro>=1.0.13
Requires-Dist: webdataset>=1.0.2
Requires-Dist: xformers>=0.0.35; sys_platform == 'linux'
Description-Content-Type: text/markdown

# MLV-Bench: Human vs. Machine Mid-Level Vision

This repository contains the code for the paper "[Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images](https://arxiv.org/abs/2607.08932)" and provides reusable components for testing mid-level representations in Vision Transformers.


## Prerequisites
Clone this repository and install the Python dependencies:

```bash
git clone https://github.com/mtangemann/mlvbench
cd mlvbench
uv sync
```


## Probing figure-ground organization in Vision Transformers
All experiment code is provided in [`experiments/figure_ground_shape_cues`](experiments/figure_ground_shape_cues).

```bash
cd experiments/figure_ground_shape_cues

# Train probes for an individual model and condition. Recommend for debugging and
# testing the setup.
python experiment.py --condition natural --model timm/vit_small_patch16_dinov3.lvd1689m

# Use the `launch.py` script to train probes for all networks and conditions. This
# supports submitting parallel jobs on a Slurm cluster, but you might have to adapt the
# Slurm settings for your cluster.
python launch.py path/to/output
```

## MLV-Bench
Several of the components provided by MLV-Bench are are generic and can be reused for
different experiments, including the *feature extraction* and *probe fitting*. This
components are contained in the core [`mlvbench`](mlvbench) library.

```python
from mlvbench.models import build_model
from mlvbench.trainer import ProbeTrainer

model = build_model("timm/vit_base_patch16_224.orig_in21k")


# --- Feature Extraction ---

images = ...  # torch uint8 Tensor with shape (B, 3, H, W)
features = model.forward_features(images)

features.patch("block.11")                  # Patch features with shape (B, N, C)
features.patch("block.11", format="BHWC")   # or reshaped to a spatial feature map
features.cls("block.11")                    # CLS token


# --- Probe Fitting ---

data_module = MyDataModule(...)

trainer = ProbeTrainer(
    task: "binary_segmentation",
    probe: "linear",
    data_module: data_module,
    batch_size: 256,
    max_steps: 1000,
)
trainer.fit(model)

# Access the fitted probes via `model.probes`
print(f"Fitted {len(model.probes)} probes to the following layers:")
for probe in model.probes:
    print(probe.layer)

# Evaluate probes
test_loader = data_module.test_loader(batch_size=256)
trainer.evaluate(model, test_loader, "output/example/evaluation")
```

Have a look at the [API documentation](https://mtangemann.github.io/mlvbench) for more
information.


## Citation
Please cite our paper if you use MLV-Bench for your experiments.

```bibtex
@misc{tangemann2026figureground,
      title={Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images}, 
      author={Matthias Tangemann and Benjamin Lo and Zygmunt Pizlo and Kaleem Siddiqi and Dirk B. Walther and Sven Dickinson},
      year={2026},
      eprint={2607.08932},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.08932}, 
}
```

Please also make sure to cite the original authors for all models that you use. Each
model provides a link to more information as `model.url`.
