tiny-dnn

by tiny-dnn

tiny-dnn / tiny-dnn

header only, dependency-free deep learning framework in C++14

5.2K Stars 1.3K Forks Last release: over 4 years ago (v0.1.1) Other 1.0K Commits 6 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:




Maintainers Wanted

The project may be abandoned since the maintainer(s) are just looking to move on. In the case anyone is interested in continuing the project, let us know so that we can discuss next steps.

Please visit: https://groups.google.com/forum/#!forum/tiny-dnn-dev


Join the chat at https://gitter.im/tiny-dnn/users Docs License Coverage Status

tiny-dnn is a C++14 implementation of deep learning. It is suitable for deep learning on limited computational resource, embedded systems and IoT devices.

|

Linux/Mac OS
|
Windows
| |------------------|-------------| |Build Status|Build status|

Table of contents

Check out the documentation for more info.

What's New

Features

  • Reasonably fast, without GPU:
    • With TBB threading and SSE/AVX vectorization.
    • 98.8% accuracy on MNIST in 13 minutes training (@Core i7-3520M).
  • Portable & header-only:
    • Runs anywhere as long as you have a compiler which supports C++14.
    • Just include tiny_dnn.h and write your model in C++. There is nothing to install.
  • Easy to integrate with real applications:
    • No output to stdout/stderr.
    • A constant throughput (simple parallelization model, no garbage collection).
    • Works without throwing an exception.
    • Can import caffe's model.
  • Simply implemented:
    • A good library for learning neural networks.

Comparison with other libraries

Please see wiki page.

Supported networks

layer-types

  • core
    • fully connected
    • dropout
    • linear operation
    • zero padding
    • power
  • convolution
    • convolutional
    • average pooling
    • max pooling
    • deconvolutional
    • average unpooling
    • max unpooling
  • normalization
    • contrast normalization (only forward pass)
    • batch normalization
  • split/merge
    • concat
    • slice
    • elementwise-add

activation functions

  • tanh
  • asinh
  • sigmoid
  • softmax
  • softplus
  • softsign
  • rectified linear(relu)
  • leaky relu
  • identity
  • scaled tanh
  • exponential linear units(elu)
  • scaled exponential linear units (selu)

loss functions

  • cross-entropy
  • mean squared error
  • mean absolute error
  • mean absolute error with epsilon range

optimization algorithms

  • stochastic gradient descent (with/without L2 normalization)
  • momentum and Nesterov momentum
  • adagrad
  • rmsprop
  • adam
  • adamax

Dependencies

Nothing. All you need is a C++14 compiler (gcc 4.9+, clang 3.6+ or VS 2015+).

Build

tiny-dnn is header-only, so there's nothing to build. If you want to execute sample program or unit tests, you need to install cmake and type the following commands:

cmake . -DBUILD_EXAMPLES=ON
make

Then change to

examples
directory and run executable files.

If you would like to use IDE like Visual Studio or Xcode, you can also use cmake to generate corresponding files:

cmake . -G "Xcode"            # for Xcode users
cmake . -G "NMake Makefiles"  # for Windows Visual Studio users

Then open .sln file in visual studio and build(on windows/msvc), or type

make
command(on linux/mac/windows-mingw).

Some cmake options are available:

|options|description|default|additional requirements to use| |-----|-----|----|----| |USETBB|Use Intel TBB for parallelization|OFF1|Intel TBB| |USEOMP|Use OpenMP for parallelization|OFF1|OpenMP Compiler| |USESSE|Use Intel SSE instruction set|ON|Intel CPU which supports SSE| |USEAVX|Use Intel AVX instruction set|ON|Intel CPU which supports AVX| |USEAVX2|Build tiny-dnn with AVX2 library support|OFF|Intel CPU which supports AVX2| |USENNPACK|Use NNPACK for convolution operation|OFF|Acceleration package for neural networks on multi-core CPUs| |USEOPENCL|Enable/Disable OpenCL support (experimental)|OFF|The open standard for parallel programming of heterogeneous systems| |USELIBDNN|Use Greentea LibDNN for convolution operation with GPU via OpenCL (experimental)|OFF|An universal convolution implementation supporting CUDA and OpenCL| |USESERIALIZER|Enable model serialization|ON2|-| |USEDOUBLE|Use double precision computations instead of single precision|OFF|-| |USEASAN|Use Address Sanitizer|OFF|clang or gcc compiler| |USEIMAGEAPI|Enable Image API support|ON|-| |USEGEMMLOWP|Enable gemmlowp support|OFF|-| |BUILDTESTS|Build unit tests|OFF3|-| |BUILDEXAMPLES|Build example projects|OFF|-| |BUILD_DOCS|Build documentation|OFF|Doxygen| |PROFILE|Build unit tests|OFF|gprof|

1 tiny-dnn use C++14 standard library for parallelization by default.

2 If you don't use serialization, you can switch off to speedup compilation time.

3 tiny-dnn uses Google Test as default framework to run unit tests. No pre-installation required, it's automatically downloaded during CMake configuration.

For example, type the following commands if you want to use Intel TBB and build tests:

bash
cmake -DUSE_TBB=ON -DBUILD_TESTS=ON .

Customize configurations

You can edit include/config.h to customize default behavior.

Examples

Construct convolutional neural networks

#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
using namespace tiny_dnn::layers;

void construct_cnn() { using namespace tiny_dnn;

network<sequential> net;

// add layers
net &lt;&lt; conv(32, 32, 5, 1, 6) &lt;&lt; tanh()  // in:32x32x1, 5x5conv, 6fmaps
    &lt;&lt; ave_pool(28, 28, 6, 2) &lt;&lt; tanh() // in:28x28x6, 2x2pooling
    &lt;&lt; fc(14 * 14 * 6, 120) &lt;&lt; tanh()   // in:14x14x6, out:120
    &lt;&lt; fc(120, 10);                     // in:120,     out:10

assert(net.in_data_size() == 32 * 32);
assert(net.out_data_size() == 10);

// load MNIST dataset
std::vector<label_t> train_labels;
std::vector<vec_t> train_images;

parse_mnist_labels("train-labels.idx1-ubyte", &amp;train_labels);
parse_mnist_images("train-images.idx3-ubyte", &amp;train_images, -1.0, 1.0, 2, 2);

// declare optimization algorithm
adagrad optimizer;

// train (50-epoch, 30-minibatch)
net.train<mse adagrad>(optimizer, train_images, train_labels, 30, 50);

// save
net.save("net");

// load
// network<sequential> net2;
// net2.load("net");

}

Construct multi-layer perceptron (mlp)

#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
using namespace tiny_dnn::layers;

void construct_mlp() { network net;

net &lt;&lt; fc(32 * 32, 300) &lt;&lt; sigmoid() &lt;&lt; fc(300, 10);

assert(net.in_data_size() == 32 * 32);
assert(net.out_data_size() == 10);

}

Another way to construct mlp

#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;

void construct_mlp() { auto mynet = make_mlp({ 32 * 32, 300, 10 });

assert(mynet.in_data_size() == 32 * 32);
assert(mynet.out_data_size() == 10);

}

For more samples, read examples/main.cpp or MNIST example page.

Contributing

Since deep learning community is rapidly growing, we'd love to get contributions from you to accelerate tiny-dnn development! For a quick guide to contributing, take a look at the Contribution Documents.

References

[1] Y. Bengio, Practical Recommendations for Gradient-Based Training of Deep Architectures. arXiv:1206.5533v2, 2012

[2] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278-2324.

Other useful reference lists: - UFLDL Recommended Readings - deeplearning.net reading list

License

The BSD 3-Clause License

Gitter rooms

We have gitter rooms for discussing new features & QA. Feel free to join us!

developers https://gitter.im/tiny-dnn/developers
users https://gitter.im/tiny-dnn/users

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.