Need help with Group-Normalization-Tensorflow?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

208 Stars 35 Forks MIT License 18 Commits 7 Opened issues


A TensorFlow implementation of Group Normalization on the task of image classification

Services available


Need anything else?

Contributors list

# 62,728
15 commits
# 206,405
1 commit

Group Normalization

As part of the implementation series of Joseph Lim's group at USC, our motivation is to accelerate (or sometimes delay) research in the AI community by promoting open-source projects. To this end, we implement state-of-the-art research papers, and publicly share them with concise reports. Please visit our group github site for other projects.

This project is implemented by Shao-Hua Sun and the codes have been reviewed by Te-Lin Wu before being published.


This project includes a Tensorflow implementation of Group Normalizations proposed in the paper Group Normalization by Wu et al. Batch Normalization (BN) has been widely employed in the trainings of deep neural networks to alleviate the internal covariate shift [1].Specifically, BN aims to transform the inputs of each layer in such a way that they have a mean output activation of zero and standard deviation of one. While BN demonstrates it effectiveness in a variety of fields including computer vision, natural language processing, speech processing, robotics, etc., BN's performance substantially decrease when the training batch size become smaller, which limits the gain of utilizing BN in a task requiring small batches constrained by memory consumption.

Motivated by this phenomenon, the Group Normalization (GN) technique is proposed. Instead of normalizing along the batch dimension, GN divides the channels into groups and computes within each group the mean and variance. Therefore, GN’s computation is independent of batch sizes, and so does its accuracy. The experiment section of the paper demonstrates the effectiveness of GN in a wide range of visual tasks, which include image classification (ImageNet), object detection and segmentation (COCO), and video classification (Kinect). This repository is simply a toy repository for those who want to quickly test GN and compare it against BN.

The illustration from the original GN paper. Each cube represent a 4D tensor of feature maps. Note that the spatial dimension are combined as a single dimension for visualization. N denotes the batch axis, C denotes the channel axis, and H, W denote the spatial axes. The values in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels.

Based on the implementation of this repository, GN is around 20% slower than BN on datasets such as CIFAR-10 and SVHN, which is probably because of the extra reshape and transpose operations. However, when the network goes deeper and the number of channels increase, GN gets even slower due to a larger group size. The model uses GN is around 4 times slower than the one uses BN when being trained ImageNet. This is not reported in the original GN paper.

*This code is still being developed and subject to change.




Train models on MNIST, Fashion MNIST, SVHN, CIFAR-10 datasets: - Download datasets with:

$ python --dataset MNIST Fashion SVHN CIFAR10
Train models on Tiny ImageNet - Download the dataset from the webpage. - Move the downloaded file (named ) to
and unzip it.

Train models on ImageNet - The ImageNet dataset is located in the Downloads section of the website. Please specify the path to the downloaded dataset by changing the variable

. Also, please provide a list of file names for trainings in the directory
with the file name
. By default, the
includes all the training images in ImageNet dataset.

Train models with downloaded datasets:

Specify the type of normalization you want to use by

--norm_type batch
--norm_type group
and specify the batch size with
--batch_size BATCH_SIZE
$ python --dataset MNIST --learning_rate 1e-3
$ python --dataset Fashion --prefix test
$ python --dataset SVHN --batch_size 128
$ python --dataset CIFAR10 

Train and test your own datasets:

  • Create a directory

    $ mkdir datasets/YOUR_DATASET
  • Store your data as an h5py file datasets/YOUR_DATASET/data.hy and each data point contains

    • 'image': has shape [h, w, c], where c is the number of channels (grayscale images: 1, color images: 3)
    • 'label': represented as an one-hot vector
  • Maintain a list datasets/YOUR_DATASET/id.txt listing ids of all data points

  • Modify including args, data_info, etc.

  • Finally, train and test models:

    $ python --dataset YOUR_DATASET
    $ python --dataset YOUR_DATASET



| Color | Batch Size | | :------- | ---------- | | Orange | 1 | | Blue | 2 | | Sky blue | 4 | | Red | 8 | | Green | 16 | | Pink | 32 |

  • Loss

  • Accuracy


| Color | Batch Size | | :------- | ---------- | | Pink | 1 | | Blue | 2 | | Sky blue | 4 | | Green | 8 | | Red | 16 | | Orange | 32 |

  • Loss

  • Accuracy


The trainings are ongoing...

| Color | Norm Type | | :----- | ------------------- | | Orange | Group Normalization | | Blue | Batch Normalization |


The Group Normalization divides the channels into groups and computes within each group the mean and variance, and therefore its performance independent of training batch sizes, which is verified with this implementation. However, the performance of Batch Normalization does not vary a lot with different batch sizes on smaller image datasets including CIFAR-10, SVHN, etc. The ImageNet experiments are ongoing and the results will be updated later.

Related works


Shao-Hua Sun / @shaohua0116 @ Joseph Lim's research lab @ USC

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.