Need help with PlaneNet?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

335 Stars 76 Forks MIT License 237 Commits 19 Opened issues


PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image

Services available


Need anything else?

Contributors list

# 37,803
154 commits
# 253,982
37 commits

PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image

By Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa


This paper presents the first end-to-end neural architecture for piece-wise planar reconstruction from a single RGB image. The proposed network, PlaneNet, learns to directly infer a set of plane parameters and corresponding plane segmentation masks. For more details, please refer to our CVPR 2018 paper or visit our project website.


We developed a better technique, PlaneRCNN, for piece-wise planar detection as described in our recent arXiv paper. Unfortunately, we cannot release the code and data yet.

We add script for extracting plane information from the original ScanNet dataset and rendering 3D planar segmentation results to 2D views. Please see the README in folder data_preparation/ for details. Note that we made some modifications to the heuristic-heavy plane fitting algorithms when cleaning up the messy codes developed over time. So the plane fitting results will be slightly different with the training data we used (provided in the .tfrecords files).

PyTorch training and testing codes are available now (still experimental and without the CRF module).


Python 2.7, TensorFlow (>= 1.3), numpy, opencv 3.

Getting started


Please run the following commands to compile the library for the crfasrnn module.

cd cpp
cd ..

To train the network, you also need to run the following commands to compile the library for computing the set matching loss. You need Eigen (I am using Eigen 3.2.92) for the compilation. (Please see here for details.)

cd nndistance
cd ..

Data preparation

We convert ScanNet data to .tfrecords files for training and testing. The training data can be downloaded from here (or here if you cannont access the previous one), and the validation data can be downloaded from here (or here).

If you download the training data from the BOX link, please run the following command to merge downloaded files into one .tfrecords file.

cat training_data_segments/* > planes_scannet_train.tfrecords


To train the network from the pretrained DeepLab network, please first download the DeepLab model here (under the Caffe to TensorFlow conversion), and then run the following command.

python --restore=0 --modelPathDeepLab="path to the deep lab model" --dataFolder="folder which contains tfrecords files"


Please first download our trained network from here (or here) and put the uncompressed folder under ./checkpoint folder.

To evaluate the performance against existing methods, please run:

python --dataFolder="folder which contains tfrecords files"

Plane representation

A plane is represented by three parameters and a segmentation mask. If the plane equation is nx=d where n is the surface normal and d is the plane offset, then plane parameters are nd. The plane equation is in the camera frame, where x points to the right, y points to the front, and z points to the up.


Please first download our trained network (see Evaluation section for details). Script predicts and visualizes custom images (if "customImageFolder" is specified) or ScanNet testing images (if "dataFolder" is specified).

python --customImageFolder="folder which contains custom images"
python --dataFolder="folder which contains tfrecords files" [--startIndex=0] [--numImages=30]

This will generate visualization images, a webpage containing all the visualization, as well as cache files under folder "predict/".

Same commands can be used for various applications by providing optional arguments, applicationType, imageIndex, textureImageFilename, and some application-specific arguments. The following commands are used to generate visualizations in the submission. (The TV application needs more manual specification for better visualization.)

python --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/CVPR.jpg --imageIndex=118 --applicationType=logo_texture --startIndex=118 --numImages=1
python --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/CVPR.jpg --imageIndex=118 --applicationType=logo_video --startIndex=118 --numImages=1
python --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/checkerboard.jpg --imageIndex=72 --applicationType=wall_texture --wallIndices=7,9 --startIndex=72 --numImages=1
python --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/checkerboard.jpg --imageIndex=72 --applicationType=wall_video --wallIndices=7,9 --startIndex=72 --numImages=1
python --customImageFolder=my_images/TV/ --textureImageFilename=texture_images/TV.mp4 --imageIndex=0 --applicationType=TV --wallIndices=2,9
python --customImageFolder=my_images/ruler --textureImageFilename=texture_images/ruler_36.png --imageIndex=0 --applicationType=ruler --startPixel=950,444 --endPixel=1120,2220

Note that, the above script generate image sequences for video applications. Please run the following command under the image sequence folder to generate a video:

ffmpeg -r 60 -f image2 -s 640x480 -i %04d.png -vcodec libx264 -crf 25 -pix_fmt yuv420p video.mp4

To check out the pool ball application, please run the following commands.

python --customImageFolder=my_images/pool --imageIndex=0 --applicationType=pool --estimateFocalLength=False
cd pool

Use mouse to play:)


If you have any questions, please contact me at [email protected]

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.