TensorNet

by Bihaqo

Bihaqo / TensorNet
125 Stars 34 Forks Last release: Not found 29 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

TensorNet

This is a MATLAB and Theano+Lasagne implementation of the Tensor Train layer (TT-layer) of a neural network. For a TensorFlow implementation see a separate repository.

In short, the TT-layer acts as a fully-connected layer but is much more compact and allows to use lots of hidden units without slowing down the learning and inference.
For the additional information see the following paper:

Tensorizing Neural Networks
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov; In Advances in Neural Information Processing Systems 28 (NIPS-2015) [arXiv].

Please cite it if you write a scientific paper using this code.
In BiBTeX format:

latex
@incollection{novikov15tensornet,
  author    = {Novikov, Alexander and Podoprikhin, Dmitry and Osokin, Anton and Vetrov, Dmitry},
  title     = {Tensorizing Neural Networks},
  booktitle = {Advances in Neural Information Processing Systems 28 (NIPS)},
  year      = {2015},
}

Installation

MATLAB version

Install the TT-Toolbox (just download it and run

setup.m
to add everything important into the MATLAB path).

Install the MatConvNet framework (preferably with the GPU support). TensorNet works with MatConvNet 1.0-beta11 (April 2015) and higher (the latest tested version is 1.0-beta14).
Add the

mataconvnet_path/examples
folder to the MATLAB path to be able to use the
cnn_train
function.

Copy this repository and add the

src/matlab
folder into the MATLAB path.

Now you can test TensorNet using the command

matlab
vl_test_ttlayers

To test GPU support (if you have compiled MatConvNet in GPU mode) use:

matlab
vl_test_ttlayers(1)

Theano+Lasagne version

Install fresh version of Theano and Lasagne.

Copy this repository and add the

src/python
folder into the Python path.

Pretrained models

MNIST shapes

In this experiment we compared how shapes and ranks influence the performance of the TT-layer using the MNIST dataset (see figure 1 and section 6.1 of the original paper for the details). Download models in the MatConvNet format (.mat file, 2.9 Mb) and preprocessed MNIST dataset (.mat file, 132 Mb).

You will find a cell array of models with metadata, the first and the last epochs of training included for each model. Example of usage (computing the validation error):

matlab
imdb = load('imdb.mat');
load('mnist_shapes.mat');
% Choose (for example) the 5-th model whose shape equal 4 x 8 x 8 x 4.
net = models{5}.lastEpoch.net;
% Remove the softmax layer (unnecessary during the validation).
net.layers(end) = [];
valIdx = find(imdb.images.set == 3);
res = vl_simplenn(net, imdb.images.data(:, :, :, valIdx));
scores = squeeze(res(end).x);
[bestScore, best] = max(scores);
acc = mean(best == imdb.images.labels(valIdx));
fprintf('Accuracy is %f\n', acc);

Reproducing experiments

Right now just one basic example on the MNIST dataset is available (more experiments from the paper are coming soon). To try it out, navigate to the

experiments/mnist
folder and type the following command in the MATLAB prompt:
matlab
[net_tt, info_tt] = cnn_mnist_tt('expDir', 'data/mnist-tt');

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.