Need help with MultiNet?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

492 Stars 239 Forks MIT License 52 Commits 32 Opened issues


Real-time Joint Semantic Reasoning for Autonomous Driving

Services available


Need anything else?

Contributors list

# 17,052
47 commits
# 110,571
1 commit
# 306,590
1 commit
# 306,804
1 commit
# 12,698
1 commit


MultiNet is able to jointly perform road segmentation, car detection and street classification. The model achieves real-time speed and state-of-the-art performance in segmentation. Check out our paper for a detailed model description.

MultiNet is optimized to perform well at a real-time speed. It has two components: KittiSeg, which sets a new state-of-the art in road segmentation; and KittiBox, which improves over the baseline Faster-RCNN in both inference speed and detection performance.

The model is designed as an encoder-decoder architecture. It utilizes one VGG encoder and several independent decoders for each task. This repository contains generic code that combines several tensorflow models in one network. The code for the individual tasks is provided by the KittiSeg, KittiBox, and KittiClass repositories. These repositories are utilized as submodules in this project. This project is built to be compatible with the TensorVision back end, which allows for organizing experiments in a very clean way.


The code requires Python 2.7, Tensorflow 1.0, as well as the following python libraries:

  • matplotlib
  • numpy
  • Pillow
  • scipy
  • runcython
  • commentjson

Those modules can be installed using:

pip install numpy scipy pillow matplotlib runcython commentjson
pip install -r requirements.txt


  1. Clone this repository:
  2. Initialize all submodules:
    git submodule update --init --recursive
  3. cd submodules/KittiBox/submodules/utils/ && make
    to build cython code
  4. [Optional] Download Kitti Road Data:
    1. Retrieve kitti data url here:
    2. Call
      python --kitti_url URL_YOU_RETRIEVED
  5. [Optional] Run
    cd submodules/KittiBox/submodules/KittiObjective2/ && make
    to build the Kitti evaluation code (see submodules/KittiBox/submodules/KittiObjective2/ for more information)

Running the model using
only requires you to perform step 1-3. Step 4 and 5 is only required if you want to train your own model using
. Note that I recommend using
instead of downloading the data yourself. The script will also extract and prepare the data. See Section Manage data storage if you like to control where the data is stored.
To update MultiNet do:
  1. Pull all patches:
    git pull
  2. Update all submodules:
    git submodule update --init --recursive

If you forget the second step you might end up with an inconstant repository state. You will already have the new code for MultiNet but run it old submodule versions code. This can work, but I do not run any tests to verify this.


Getting started


python --gpus 0 --input data/demo/um_000005.png
to obtain a prediction using demo.png as input.


to evaluate a trained model.


python --hypes hypes/multinet2.json
to train a multinet2

If you like to understand the code, I would recommend looking at first. I have documented each step as thoroughly as possible in this file.

Only training of MultiNet3 (joint detection and segmentation) is supported out of the box. The data to train the classification model is not public an those cannot be used to train the full MultiNet3 (detection, segmentation and classification). The full code is given here, so you can still train MultiNet3 if you have your own data.

Manage Data Storage

MultiNet allows to separate data storage from code. This is very useful in many server environments. By default, the data is stored in the folder

and the output of runs in
. This behaviour can be changed by setting the bash environment variables:


in your
and the all data will be downloaded to
. Include
in your
and all runs will be saved to

Modifying Model & Train on your own data

The model is controlled by the file

. This file points the code to the implementation of the submodels. The MultiNet code then loads all models provided and integrates the decoders into one neural network. To train on your own data, it should be enough to modify the hype files of the submodels. A good start will be the KittiSeg model, which is very well documented.
    "models": {
        "segmentation" : "../submodules/KittiSeg/hypes/KittiSeg.json",
        "detection" : "../submodules/KittiBox/hypes/kittiBox.json",
        "road" : "../submodules/KittiClass/hypes/KittiClass.json"

RUNDIR and Experiment Organization

MultiNet helps you to organize a large number of experiments. To do so, the output of each run is stored in its own rundir. Each rundir contains:

  • output.log
    a copy of the training output which was printed to your screen
  • tensorflow events
    tensorboard can be run in rundir
  • tensorflow checkpoints
    the trained model can be loaded from rundir
  • [dir] images
    a folder containing example output images.
    controls how often the whole validation set is dumped
  • [dir] model_files
    A copy of all source code need to build the model. This can be very useful of you have many versions of the model.

To keep track of all the experiments, you can give each rundir a unique name with the

flag. The
flag will store the run in a separate subfolder allowing to run different series of experiments. As an example,
python --project batch_size_bench --name size_5
will use the following dir as rundir:

The flag

is very useful to not spam your rundir.

Useful Flags & Variabels

Here are some Flags which will be useful when working with KittiSeg and TensorVision. All flags are available across all scripts.

: specify which hype-file to use
: specify which logdir to use
: specify on which GPUs to run the code
: assign a name to the run
: assign a project to the run
: debug run, logdir will be set to

In addition the following TensorVision environment Variables will be useful:

: specify meta directory for data
: specify meta directory for output
: specify default GPU behaviour.

On a cluster it is useful to set

. This will make the flag
mandatory and ensure, that run will be executed on the right GPU.


If you benefit from this code, please cite our paper:

  title={MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving},
  author={Teichmann, Marvin and Weber, Michael and Zoellner, Marius and Cipolla, Roberto and Urtasun, Raquel},
  journal={arXiv preprint arXiv:1612.07695},

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.