Metadata-Version: 2.1
Name: intrinsic-dimensionality
Version: 0.1
Summary: Dense and fastfood transform wrappers to reproduce "Intrinsic dimensionality of objective landscapes" by Li et al. (2018)
Home-page: https://github.com/jgamper/intrinsic-dimensionality
Author: Jevgenij Gamper
Author-email: jevgenij.gamper5@gmail.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.6
Description-Content-Type: text/markdown
Requires-Dist: backcall (==0.2.0)
Requires-Dist: cycler (==0.10.0)
Requires-Dist: decorator (==4.4.2)
Requires-Dist: future (==0.18.2)
Requires-Dist: ipykernel (==5.3.3)
Requires-Dist: ipython (==7.16.1)
Requires-Dist: ipython-genutils (==0.2.0)
Requires-Dist: jedi (==0.17.2)
Requires-Dist: jupyter-client (==6.1.6)
Requires-Dist: jupyter-core (==4.6.3)
Requires-Dist: kiwisolver (==1.2.0)
Requires-Dist: matplotlib (==3.3.0)
Requires-Dist: numpy (==1.19.1)
Requires-Dist: pandas (==1.0.5)
Requires-Dist: parso (==0.7.0)
Requires-Dist: pexpect (==4.8.0)
Requires-Dist: pickleshare (==0.7.5)
Requires-Dist: Pillow (==7.2.0)
Requires-Dist: prompt-toolkit (==3.0.5)
Requires-Dist: ptyprocess (==0.6.0)
Requires-Dist: Pygments (==2.6.1)
Requires-Dist: pyparsing (==2.4.7)
Requires-Dist: python-dateutil (==2.8.1)
Requires-Dist: pytz (==2020.1)
Requires-Dist: pyzmq (==19.0.1)
Requires-Dist: scipy (==1.5.1)
Requires-Dist: seaborn (==0.10.1)
Requires-Dist: six (==1.15.0)
Requires-Dist: torch (==1.5.1)
Requires-Dist: torchvision (==0.6.1)
Requires-Dist: tornado (==6.0.4)
Requires-Dist: tqdm (==4.48.0)
Requires-Dist: traitlets (==4.3.3)
Requires-Dist: wcwidth (==0.2.5)

# About

This package includes fastfood and dense transformation wrappers for pytorch modules, primarily to reproduce results from
[Li, Chunyuan, et al. "Measuring the intrinsic dimension of objective landscapes." arXiv preprint arXiv:1804.08838 (2018)](https://arxiv.org/abs/1804.08838) - see below for info.

* All contributions are welcome! Please raise an issue for a bug, feature or pull request!

* Give this repo a star! :star:

<p align="center">
    <img src="https://raw.githubusercontent.com/jgamper/intrinsic-dimensionality/master/assets/intrinsic_star.png" width="600"/>
<p>

# Install

`pip install intrinsic-dimensionality`

# Quick start on your classification task!

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = DEVICE_NUM
import torch
from torch import nn
import torchvision.models as models
from intrinsic import FastFoodWrap

class Classifier(nn.Module):
    def __init__(self, input_dim, n_classes):
        super(Classifier, self).__init__()
        self.fc = nn.Linear(input_dim, n_classes)
        self.maxpool = nn.AdaptiveMaxPool2d(1)

    def forward(self, x):
        x = self.maxpool(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)
        return x

def get_resnet(encoder_name, num_classes, pretrained=False):
    assert encoder_name in ["resnet18", "resnet50"], "{} is a wrong encoder name!".format(encoder_name)
    if encoder_name == "resnet18":
        model = models.resnet18(pretrained=pretrained)
        latent_dim = 512
    else:
        model = models.resnet50(pretrained=pretrained)
        latent_dim = 2048
    children = (list(model.children())[:-2] + [Classifier(latent_dim, num_classes)])
    model = torch.nn.Sequential(*children)
    return model

# Get model and wrap it in fastfood
model = get_resnet("resnet18", num_classes=YOUR_NUMBER_OF_CLASSES).cuda()
model = FastFoodWrap(model, intrinsic_dimension=100, device=DEVICE_NUM)
```

# Reproducing experiments from the paper

Full thread about reproducibility results is available [here](https://twitter.com/brutforcimag/status/1240335205807816705?s=20).
Note that some hyper-parameters were not listed in the paper - I raised issues on Uber's Github repo [here](https://github.com/uber-research/intrinsic-dimension/issues/5).

I am able to reproduce their MNIST results with LR=0.0003, batch size 32 for both dense and fastfood transformations
using FCN (fcn-dense, fcn-fastfood). However, not for LeNet (cnn-dense, cnn-fastfood).

<p align="center">
    <img src="https://raw.githubusercontent.com/jgamper/intrinsic-dimensionality/master/assets/mnist_reproduction.png" width="600"/>
<p>

For CIFAR-10, with far larger resnet (Resnet-18 11mil param) vs 280k 20-layer resnet used in the paper,
results appear to be similar. FCN results in appendix (Fig S7) suggest some variation is to be expected.

<p align="center">
    <img src="https://raw.githubusercontent.com/jgamper/intrinsic-dimensionality/master/assets/cifar10.png" width="600"/>
<p>

# Cite

```
@misc{jgamper2020intrinsic,
  title   = "Intrinsic-dimensionality Pytorch",
  author  = "Gamper, Jevgenij",
  year    = "2020",
  url     = "https://github.com/jgamper/intrinsic-dimensionality"
}

@article{li2018measuring,
  title={Measuring the intrinsic dimension of objective landscapes},
  author={Li, Chunyuan and Farkhoor, Heerad and Liu, Rosanne and Yosinski, Jason},
  journal={arXiv preprint arXiv:1804.08838},
  year={2018}
}
```


