Mirror

Pytorch CNN Visualisation Tool

Francesco Saverio Zuppichini

This is a raw beta so expect lots of things to change and improve over time.

An interactive version of this tutorial can be found here

Getting started

To install mirror run

pip install git+https://github.com/FrancescoSaverioZuppichini/mirror.git

Getting Started

A basic example

from mirror import mirror
from mirror.visualisations.web import *
from PIL import Image
from torchvision.models import resnet101, resnet18, vgg16, alexnet
from torchvision.transforms import ToTensor, Resize, Compose

create a model
model = vgg16(pretrained=True)
open some images
cat = Image.open("./cat.jpg")
dog_and_cat = Image.open("./dog_and_cat.jpg")
resize the image and make it a tensor
to_input = Compose([Resize((224, 224)), ToTensor()])
call mirror with the inputs and the model
mirror([to_input(cat), to_input(dog_and_cat)], model, visualisations=[BackProp, GradCam, DeepDream])

 * Serving Flask app "mirror.App" (lazy loading)


Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
Debug mode: off


Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

It will automatic open a new tab in your browser

On the left you can see your model tree structure, by clicking on one layer all his children are showed. On the right there are the visualisation settings. You can select your input by clicking on the bottom tab.

Available Visualisations

All visualisation available for the web app are inside

.mirror.visualisations.web

Weights

Deep Dream

Back Prop / Guide Back Prop

By clicking on the radio button 'guide', all the relus negative output will be set to zero producing a nicer looking image

Grad Cam / Guide Grad Cam

Using with Tensors

If you want, you can use the vanilla version of each visualisation by importing them from

.mirror.visualisation.core

from mirror.visualisations.core import GradCam

create a model
model = vgg16(pretrained=True)
open some images
cat = Image.open("./cat.jpg")
dog_and_cat = Image.open("./dog_and_cat.jpg")
resize the image and make it a tensor
to_input = Compose([Resize((224, 224)), ToTensor()])
cam = GradCam(model, device='cpu')
cam(to_input(cat).unsqueeze(0), None) # will return the output image and some additional information

Create a Visualisation

To create a visualisation you first have to subclass the

Visualisation

__call__

repeat

from mirror.visualisations.core import Visualisation

class RepeatInput(Visualisation):
def __call__(self, inputs, layer, repeat=1):
    return inputs.repeat(repeat, 1, 1, 1), None

This class repeats the input for

repeat

Connect to the web interface

To connect our fancy visualisation to the web interface, we have to create a

WebInterface

WebInterface.from_visualisation

It follows and example

from mirror.visualisations.web import WebInterface
from functools import partial

params = {'repeat' : {
                     'type' : 'slider',
                     'min' : 1,
                     'max' : 100,
                     'value' : 2,
                     'step': 1,
                     'params': {}
                 }
        }
visualisation = partial(WebInterface.from_visualisation, RepeatInput, params=params, name='Repeat')

First we import

WebInterface

partial

RepeatInput

repeat

{ 'repeat' : { ... }' }

The value of that dictionary is the configuration for one of the basic UI elements: slider, textfield and radio.

The input is stored in the

value

Then we call

WebInterface.from_visualisation

partial

mirror

The final result is

Change the front-end

All the front-end is developed usin React and Material-UI, two very known frameworks, making easier for anybody to contribuite.

You can customise the front-end by changing the source code in

mirror/client

I was not able to serve the static file directly from the /mirror/client/build folder if you know how to do it any pull request is welcome :)

cd ./mirror/mirror/client // assuming the root folder is called mirror
npm run build

Then you need to move the fiels from the

mirror/mirror/client/build

mirror/mirror

mirror/mirro/static

mv ./build/static ../ && cp ./build/* ../static/

TODO

• [x] Cache reused layer
• [x] Make a generic abstraction of a visualisation in order to add more features
  
• [x] Add dropdown as parameter
• [x] Add text field
• [x] Support multiple inputs
• [x] Support multiple models

• Add all visualisation present here https://github.com/utkuozbulak/pytorch-cnn-visualizations

  • [x] Gradient visualization with vanilla backpropagation
  • [x] Gradient visualization with guided backpropagation [1]
  • [x] Gradient visualization with saliency maps [4]
  • [x] CNN filter visualization [9]
  • [x] Deep dream [10]
  • [ ] Class specific image generation [4]
  • [x] Grad Cam

• [ ] Add a

  output_transformation

  params for each visualisation to allow better customisation

• [ ] Add a

  input_transformation

  params for each visualisation to allow better customisation