Need help with morb?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

benanne
175 Stars 44 Forks GNU General Public License v3.0 140 Commits 4 Opened issues

Description

Modular Restricted Boltzmann Machine (RBM) implementation using Theano

Services available

!
?

Need anything else?

Contributors list

# 26,203
deep-le...
theano
Python
TeX
136 commits

Morb: a modular RBM implementation in Theano

Morb logo

Introduction

Morb is a toolbox for building and training Restricted Boltzmann Machine models in Theano. It is intended to be modular, so that a variety of different models can be built from their elementary parts. A second goal is for it to be extensible, so that new algorithms and techniques can be plugged in easily.

The elementary parts in question are different types of units, which can be connected with different types of parameters. A schematic diagram of the architecture can be viewed below.

A unit type defines the distribution of that unit. For example, binary units are Bernoulli distributed. Several unit types are available, and new ones can be defined easily.

The different types of parameters form the trainable part of the model. These include biases, regular weights, convolutional weights and third order weights, amongst others. New parameter types can be defined by specifying the terms they contribute to the activations of each of the units they tie, the term they contribute to the model energy function, and the gradient of the energy function with respect to the parameters.

To train the model, one has to specify how the parameters should be updated in each step of the training process. This is possible by defining updaters, which can be composed. For example, one can combine a contrastive divergence updater with a weight decay updater and a sparsity regularisation updater. Momentum can also be applied with a momentum updater, which encapsulates another updater. Some updaters, like the contrastive divergence updater, calculate parameter updates from statistics obtained from training data.

Finally, a trainer is used to compile the symbolical parameter update expressions into a training function.

Schematic diagram of Morb's RBM architecture

Example

Below is a simple example, in which an RBM with binary visibles and binary hiddens is trained on an unspecified dataset using one-step contrastive divergence (CD-1), with some weight decay.

from morb import base, units, parameters, stats, updaters, trainers, monitors
import numpy
import theano.tensor as T

define hyperparameters

learning_rate = 0.01 weight_decay = 0.02 minibatch_size = 32 epochs = 50

load dataset

data = ...

construct RBM model

rbm = base.RBM()

rbm.v = units.BinaryUnits(rbm) # visibles rbm.h = units.BinaryUnits(rbm) # hiddens

rbm.W = parameters.ProdParameters(rbm, [rbm.v, rbm.h], initial_W) # weights rbm.bv = parameters.BiasParameters(rbm, rbm.v, initial_bv) # visible bias rbm.bh = parameters.BiasParameters(rbm, rbm.h, initial_bh) # hidden bias

define a variable map, that maps the 'input' units to Theano variables.

initial_vmap = { rbm.v: T.matrix('v') }

compute symbolic CD-1 statistics

s = stats.cd_stats(rbm, initial_vmap, visible_units=[rbm.v], hidden_units=[rbm.h], k=1)

create an updater for each parameter variable

umap = {} for variable in [rbm.W.W, rbm.bv.b, rbm.bh.b]: new_value = variable + learning_rate * (updaters.CDUpdater(rbm, variable, s) - decay * updaters.DecayUpdater(variable)) umap[variable] = new_value

monitor reconstruction cost during training

mse = monitors.reconstruction_mse(s, rbm.v)

train the model

t = trainers.MinibatchTrainer(rbm, umap) train = t.compile_function(initial_vmap, mb_size=minibatch_size, monitors=[mse])

for epoch in range(epochs): costs = [m for m in train({ rbm.v: data })] print "MSE = %.4f" % numpy.mean(costs)

Disclaimer

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.