Keras + Gaussian Processes: Learning scalable deep and recurrent kernels.
KGP extends Keras with Gaussian Process (GP) layers. It allows one to build flexible GP models with kernels structured with deep and recurrent networks built with Keras. The structured part of the model (the neural net) runs on Theano or Tensorflow. The GP layers use a custom backend based on GPML 4.0 library, and builds on KISS-GP and extensions. The models can be trained in stages or jointly, using full-batch or semi-stochastic optimization approaches (see our paper). For additional resources and tutorials on Deep Kernel Learning and KISS-GP see https://people.orie.cornell.edu/andrew/code/
KGP is compatible with: Python 2.7-3.5.
In particular, this package implements the method described in our paper:
Learning Scalable Deep Kernels with Recurrent Structure
Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, Eric P. Xing
Journal of Machine Learning Research, 2017.
KGP allows to build models in the same fashion as Keras, using the functional API. For example, a simple GP-RNN model can be built and compiled in just a few lines of code:
from keras.layers import Input, SimpleRNN from keras.optimizers import Adamfrom kgp.layers import GP from kgp.models import Model from kgp.losses import gen_gp_loss
input_shape = (10, 2) # 10 time steps, 2 dimensions batch_size = 32 nb_train_samples = 512 gp_hypers = {'lik': -2.0, 'cov': [[-0.7], [0.0]]}
Build the model
inputs = Input(shape=input_shape) rnn = SimpleRNN(32)(inputs) gp = GP(gp_hypers, batch_size=batch_size, nb_train_samples=nb_train_samples) outputs = [gp(rnn)] model = Model(inputs=inputs, outputs=outputs)
Compile the model
loss = [gen_gp_loss(gp) for gp in model.output_layers] model.compile(optimizer=Adam(1e-2), loss=loss)
Note that KGP models support arbitrary off-the-shelf optimizers from Keras.
Further resources: - A quick tutorial that walks you through the key components of the library. - A few more examples.
KGP depends on Keras and requires either Theano or TensorFlow being installed. The GPML backend requires either MATLAB or Octave and a corresponding Python interface package: Oct2Py for Octave or the MATLAB engine for Python. Generally, MATLAB backend seems to provide faster runtime. However, if you compile the latest version of Octave with JIT and OpenBLAS support, the overhead gets reduced to minimum.
If you are using Octave, you will need the
statisticspackage. You can install the package using Octave-Forge:
bash $ octave --eval "pkg install -forge -verbose io" $ octave --eval "pkg install -forge -verbose statistics"
The requirements can be installed via pip as follows (use
sudoif necessary):
$ pip install -r requirements.txt
To install the package, clone the repository and run
setup.pyas follows:
$ git clone --recursive https://github.com/alshedivat/kgp $ cd kgp $ python setup.py develop [--user]
The
--userflag (optional) will install the package for a given user only.
Note: Recursive clone is required to get GPML library as a submodule. If you already have a copy of GPML, you can set
GPML_PATHenvironment variable to point to your GPML folder instead.
Contributions and especially bug reports are more than welcome.
@article{alshedivat2017srk, title={Learning scalable deep kernels with recurrent structure}, author={Al-Shedivat, Maruan and Wilson, Andrew Gordon and Saatchi, Yunus and Hu, Zhiting and Xing, Eric P}, journal={Journal of Machine Learning Research}, volume={18}, number={1}, year={2017}, }
For questions about the code and licensing details, please contact Maruan Al-Shedivat and Andrew Gordon Wilson.