Need help with ganseeing?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

davidbau
169 Stars 17 Forks 44 Commits 2 Opened issues

Description

Seeing what a GAN cannot generate. Visualizes and quantifies object classes within scenes that are outside the range of a GAN.

Services available

!
?

Need anything else?

Contributors list

# 20,412
Python
pytorch
scene-r...
MATLAB
43 commits

Seeing What a GAN Cannot Generate

State-of-the art GANs can create increasingly realistic images, yet they are not perfect.

What is a GAN unable to generate? This repository contains the code for the ICCV 2019 paper Seeing What a GAN Cannot Generate, which introduces a framework that can be used to answer this question.

GAN reconstruction Real photo

Our goal is not to benchmark how far the generated distribution is from the target. Instead, we want to visualize and understand what is different between real and fake images.

Mode-dropping and the problem of visualizing omissions

We visualize the omissions of an image generator in two ways.

  1. We identify omissions within the distribution of images.
  2. We identify omissions within individual images.

Seeing omissions in the distribution

To understand omissions in a GAN's output distribution, we compare segmentation statistics between the GAN output and the training distribution.

A Progressive GAN trained to generate LSUN outdoor church images is analyzed below.

The model does not generate enough pixels of people, cars, fences, palm trees, or signboards compared to the training distribution. The script

run_fsd.sh
and the notebook
seeing_distributions.ipynb
show how we collect and visualize these segmentation statistics.

Seeing omissions in individual images

To understand omission in specific GAN-generated output, we must pair the output with a real photo that shows what the GAN should have drawn but did not. So we compare real training photos to a reconstructed image derived from the model of the GAN.

These visualizations are created by

run_invert.sh
.

People

As seen in the distribution statistics, thie GAN does not draw enough people. By visualizing reconstructions, we can see how: the GAN seems to avoid drawing large person figures entirely, instead synthesizing plausible scenes without people.

GAN reconstruction

Real photo

Vehicles

A similar effect is seen for vehicles.

GAN reconstruction

Real photo

Signs

GAN reconstruction

Real photo

Monuments

GAN reconstruction

Real photo
<!---
--->

Palm trees

GAN reconstruction

Real photo

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.