Perceptual Similarity Metric and Dataset [Project Page]

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. In CVPR, 2018.

Quick start

Run

pip install lpips

import lpips
loss_fn_alex = lpips.LPIPS(net='alex') # best forward scores
loss_fn_vgg = lpips.LPIPS(net='vgg') # closer to "traditional" perceptual loss, when used for optimization

import torch
img0 = torch.zeros(1,3,64,64) # image should be RGB, IMPORTANT: normalized to [-1,1]
img1 = torch.zeros(1,3,64,64)
d = loss_fn_alex(img0, img1)

More thorough information about variants is below. This repository contains our perceptual metric (LPIPS) and dataset (BAPPS). It can also be used as a "perceptual loss". This uses PyTorch; a Tensorflow alternative is here.

Table of Contents
1. Learned Perceptual Image Patch Similarity (LPIPS) metric
a. Basic Usage If you just want to run the metric through command line, this is all you need.
b. "Perceptual Loss" usage
c. About the metric
2. Berkeley-Adobe Perceptual Patch Similarity (BAPPS) dataset
a. Download
b. Evaluation
c. About the dataset
d. Train the metric using the dataset

(0) Dependencies/Setup

Installation

• Install PyTorch 1.0+ and torchvision fom http://pytorch.org

pip install -r requirements.txt

• Clone this repo:

  bash
  git clone https://github.com/richzhang/PerceptualSimilarity
  cd PerceptualSimilarity
  

(1) Learned Perceptual Image Patch Similarity (LPIPS) metric

Evaluate the distance between image patches. Higher means further/more different. Lower means more similar.

(A) Basic Usage

(A.I) Line commands

Example scripts to take the distance between 2 specific images, all corresponding pairs of images in 2 directories, or all pairs of images within a directory:

python lpips_2imgs.py -p0 imgs/ex_ref.png -p1 imgs/ex_p0.png --use_gpu
python lpips_2dirs.py -d0 imgs/ex_dir0 -d1 imgs/ex_dir1 -o imgs/example_dists.txt --use_gpu
python lpips_1dir_allpairs.py -d imgs/ex_dir_pair -o imgs/example_dists_pair.txt --use_gpu

(A.II) Python code

File test_network.py shows example usage. This snippet is all you really need.

import lpips
loss_fn = lpips.LPIPS(net='alex')
d = loss_fn.forward(im0,im1)

Variables

im0, im1

Nx3xHxW

N

HxW

[-1,+1]

d

N

Run

python test_network.py

ex_ref.png

ex_p0.png

ex_p1.png

Some Options By default in

model.initialize

net='alex'

alex

net='vgg'

lpips=True

lpips=False

(B) Backpropping through the metric

File

lpips_loss.py

python lpips_loss.py

(C) About the metric

Higher means further/more different. Lower means more similar.

We found that deep network activations work surprisingly well as a perceptual similarity metric. This was true across network architectures (SqueezeNet [2.8 MB], AlexNet [9.1 MB], and VGG [58.9 MB] provided similar scores) and supervisory signals (unsupervised, self-supervised, and supervised all perform strongly). We slightly improved scores by linearly "calibrating" networks - adding a linear layer on top of off-the-shelf classification networks. We provide 3 variants, using linear layers on top of the SqueezeNet, AlexNet (default), and VGG networks.

If you use LPIPS in your publication, please specify which version you are using. The current version is 0.1. You can set

version='0.0'

(2) Berkeley Adobe Perceptual Patch Similarity (BAPPS) dataset

(A) Downloading the dataset

Run

bash ./scripts/download_dataset.sh

./dataset

bash ./scripts/get_dataset_valonly.sh

(B) Evaluating a perceptual similarity metric on a dataset

Script

test_dataset_model.py

Dataset flags -

--dataset_mode

2afc

jnd

--datasets

--dataset_mode 2afc

train/traditional

train/cnn

val/traditional

val/cnn

val/superres

val/deblur

val/color

val/frameinterp

--dataset_mode jnd

val/traditional

val/cnn

Perceptual similarity model flags -

--model

lpips

baseline

l2

ssim

--net

squeeze

alex

vgg

net-lin

net

l2

ssim

--colorspace

Lab

RGB

l2

ssim

net-lin

net

Misc flags -

--batch_size

--use_gpu

An example usage is as follows:

python ./test_dataset_model.py --dataset_mode 2afc --datasets val/traditional val/cnn --model lpips --net alex --use_gpu --batch_size 50

(C) About the dataset

The dataset contains two types of perceptual judgements: Two Alternative Forced Choice (2AFC) and Just Noticeable Differences (JND).

(1) 2AFC Evaluators were given a patch triplet (1 reference + 2 distorted). They were asked to select which of the distorted was "closer" to the reference.

Training sets contain 2 judgments/triplet. -

train/traditional

train/cnn

train/mix

Validation sets contain 5 judgments/triplet. -

val/traditional

val/cnn

val/superres

val/deblur

val/color

val/frameinterp

Each 2AFC subdirectory contains the following folders: -

ref

p0,p1

judge

(2) JND Evaluators were presented with two patches - a reference and a distorted - for a limited time. They were asked if the patches were the same (identically) or different.

Each set contains 3 human evaluations/example. -

val/traditional

val/cnn

Each JND subdirectory contains the following folders: -

p0,p1

same

(D) Using the dataset to train the metric

See script

train_test_metric.sh

checkpoints

You can also train "scratch" and "tune" versions by running

train_test_metric_scratch.sh

train_test_metric_tune.sh

Citation

If you find this repository useful for your research, please use the following.

@inproceedings{zhang2018perceptual,
  title={The Unreasonable Effectiveness of Deep Features as a Perceptual Metric},
  author={Zhang, Richard and Isola, Phillip and Efros, Alexei A and Shechtman, Eli and Wang, Oliver},
  booktitle={CVPR},
  year={2018}
}

Acknowledgements

This repository borrows partially from the pytorch-CycleGAN-and-pix2pix repository. The average precision (AP) code is borrowed from the py-faster-rcnn repository. Angjoo Kanazawa, Connelly Barnes, Gaurav Mittal, wilhelmhb, Filippo Mameli, SuperShinyEyes, Minyoung Huh helped to improve the codebase.