Tensorflow implementation of Adversarial Autoencoders
beta1 = 0.5.
experiment/aae_mnist.py:
DATA_PATHis the path to put MNIST dataset.
SAVE_PATHis the path to save output images and trained model.
The script experiment/aae_mnist.py contains all the experiments shown here. Detailed usage for each experiment will be describe later along with the results.
--train: Train the model of Fig 1 and 3 in the paper.
--train_supervised: Train the model of Fig 6 in the paper.
--train_semisupervised: Train the model of Fig 8 in the paper.
--label: Incorporate label information in the adversarial regularization (Fig 3 in the paper).
--generate: Randomly sample images from trained model.
--viz: Visualize latent space and data manifold (only when
--ncodeis 2).
--supervise: Sampling from supervised model (Fig 6 in the paper) when
--generateis True.
--load: The epoch ID of pre-trained model to be restored.
--ncode: Dimension of code. Default:
2
--dist_type: Type of the prior distribution used to impose on the hidden codes. Default:
gaussian.
gmmfor Gaussian mixture distribution.
--noise: Add noise to encoder input (Gaussian with std=0.6).
--lr: Initial learning rate. Default:
2e-4.
--dropout: Keep probability for dropout. Default:
1.0.
--bsize: Batch size. Default:
128.
--maxepoch: Max number of epochs. Default:
100.
--encw: Weight of autoencoder loss. Default:
1.0.
--genw: Weight of z generator loss. Default:
6.0.
--disw: Weight of z discriminator loss. Default:
6.0.
--clsw: Weight of semi-supervised loss. Default:
1.0.
--ygenw: Weight of y generator loss. Default:
6.0.
--ydisw: Weight of y discriminator loss. Default:
6.0.
Architecture |
Description |
---|---|
The top row is an autoencoder. z is sampled through the re-parameterization trick discussed in variational autoencoder paper. The bottom row is a discriminator to separate samples generate from the encoder and samples from the prior distribution p(z). |
name |
value |
---|---|
Reconstruction Loss Weight | 1.0 |
Latent z G/D Loss Weight | 6.0 / 6.0 |
Batch Size | 128 |
Max Epoch | 400 |
Learning Rate | 2e-4 (initial) / 2e-5 (100 epochs) / 2e-6 (300 epochs) |
SAVE_PATH.
python aae_mnist.py --train \ --ncode CODE_DIM \ --dist_type TYPE_OF_PRIOR (`gaussian` or `gmm`)
SAVE_PATHwith name
generate_im.png.
python aae_mnist.py --generate \ --ncode CODE_DIM \ --dist_type TYPE_OF_PRIOR (`gaussian` or `gmm`)\ --load RESTORE_MODEL_ID
SAVE_PATHwith name
generate_im.pngand
latent.png. For Gaussian distribution, there will be one image for data manifold. For mixture of 10 2D Gaussian, there will be 10 images of data manifold for each component of the distribution.
python aae_mnist.py --viz \ --ncode CODE_DIM \ --dist_type TYPE_OF_PRIOR (`gaussian` or `gmm`)\ --load RESTORE_MODEL_ID<!--- name | command :--- | :--- Training |
python aae_mnist.py --train --dist_type| Random sample data |
python aae_mnist.py --generate --dist_type --load| Visualize latent space and data manifold (only when code dim = 2) |
python aae_mnist.py --viz --dist_type --load| Option |
--bsize--->
Prior Distribution |
Learned Coding Space | Learned Manifold |
---|---|---|
Architecture |
Description |
---|---|
The only difference from previous model is that the one-hot label is used as input of encoder and there is one extra class for unlabeled data. For mixture of Gaussian prior, real samples are drawn from each components for each labeled class and for unlabeled data, real samples are drawn from the mixture distribution. |
Hyperparameters are the same as previous section.
SAVE_PATH.
python aae_mnist.py --train --label\ --ncode CODE_DIM \ --dist_type TYPE_OF_PRIOR (`gaussian` or `gmm`)
Random sample data from trained model. Image will be saved in
SAVE_PATHwith name
generate_im.png.
python aae_mnist.py --generate --ncode --label --dist_type --load
Visualize latent space and data manifold (only when code dim = 2). Image will be saved in
SAVE_PATHwith name
generate_im.pngand
latent.png. For Gaussian distribution, there will be one image for data manifold. For mixture of 10 2D Gaussian, there will be 10 images of data manifold for each component of the distribution.
python aae_mnist.py --viz --label \ --ncode CODE_DIM \ --dist_type TYPE_OF_PRIOR (`gaussian` or `gmm`) \ --load RESTORE_MODEL_ID
Number of Label Used |
Learned Coding Space | Learned Manifold |
---|---|---|
Use full label | ||
10k labeled data and 40k unlabeled data |
Architecture |
Description |
---|---|
The decoder takes code as well as a one-hot vector encoding the label as input. Then it forces the network learn the code independent of the label. |
SAVE_PATH.
python aae_mnist.py --train_supervised \ --ncode CODE_DIM
SAVE_PATHwith name
sample_style.png.
python aae_mnist.py --generate --supervise\ --ncode CODE_DIM \ --load RESTORE_MODEL_ID
Code Dim=2 |
Code Dim=10 |
---|---|
Architecture |
Description |
---|---|
The encoder outputs code z as well as the estimated label y. Encoder again takes code z and one-hot label y as input. A Gaussian distribution is imposed on code z and a Categorical distribution is imposed on label y. In this implementation, the autoencoder is trained by semi-supervised classification phase every ten training steps when using 1000 label images and the one-hot label y is approximated by output of softmax. |
name |
value |
---|---|
Dimention of z | 10 |
Reconstruction Loss Weight | 1.0 |
Letant z G/D Loss Weight | 6.0 / 6.0 |
Letant y G/D Loss Weight | 6.0 / 6.0 |
Batch Size | 128 |
Max Epoch | 250 |
Learning Rate | 1e-4 (initial) / 1e-5 (150 epochs) / 1e-6 (200 epochs) |
SAVE_PATH.
python aae_mnist.py \ --ncode 10 \ --train_semisupervised \ --lr 2e-4 \ --maxepoch 250
learning curve for training set (computed only on the training set with labels)
learning curve for testing set - The accuracy on testing set is 97.10% around 200 epochs.