D3Feat repository

TensorFlow implementation of D3Feat for CVPR'2020 Oral paper "D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features", by Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan and Chiew-Lan Tai.

This paper focus on dense feature detection and description for 3D point clouds in a joint manner. If you find this project useful, please cite:

@article{bai2020d3feat,
  title={D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features},
  author={Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan and Chiew-Lan Tai},
  journal={arXiv:2003.03164 [cs.CV]},
  year={2020}
}

The PyTorch implementation can be found here.

Introduction

A successful point cloud registration often lies on robust establishment of sparse matches through discriminative 3D local features. Despite the fast evolution of learning-based 3D feature descriptors, little attention has been drawn to the learning of 3D feature detectors, even less for a joint learning of the two tasks. In this paper, we leverage a 3D fully convolutional network for 3D point clouds, and propose a novel and practical learning mechanism that densely predicts both a detection score and a description feature for each 3D point. In particular, we propose a keypoint selection strategy that overcomes the inherent density variations of 3D point clouds, and further propose a self-supervised detector loss guided by the on-the-fly feature matching results during training. Finally, our method achieves state-of-the-art results in both indoor and outdoor scenarios, evaluated on 3DMatch and KITTI datasets, and shows its strong generalization ability on the ETH dataset. Towards practical use, we show that by adopting a reliable feature detector, sampling a smaller number of features is sufficient to achieve accurate and fast point cloud alignment.

Installation

• Create the environment and install the required libaries:

     conda env create -f environment.yml
  

• Compile the customized Tensorflow operators located in

  tf_custom_ops

  . Open a terminal in this folder, and run:

    sh compile_op.sh
  

• Compile the C++ extension module for python located in

  cpp_wrappers

  . Open a terminal in this folder, and run:

    sh compile_wrappers.sh
  

The code is heavily borrowed from KPConv. You can find the guidance for compiling the tensorflow operators and C++ wrappers in INSTALL.md.

Demo

We provide a small demo to extract dense feature and detection score for two point cloud, and register them using RANSAC. The ply files are saved in the

demo_data

bash
    python demo_registration.py

demo_data

We also visualize the detected keypoints on two point cloud.

Dataset Download

3DMatch

The training set of 3DMatch[1] can be downloaded from here. It is generated by

datasets/cal_overlap.py

The test set point clouds and the evaluation files(for registration recall) can be downloaded from 3DMatch Geometric Registration website.

Please put the training set under

data/3DMatch

data/3DMatch/fragments

geometric_registration/gt_result

KITTI

The training and test set can be download from KITTI Odometry website. I follow the FCGF[3] for pre-processing.

ETH

The test set (we only use ETH dataset to evaluate the generalization ability of our method) can be downloaded from here. Detail instructions can be found in PerfectMatch[2].

Instructions to training and testing

3DMatch

The training on 3DMatch dataset can be done by running

bash
python training_3dmatch.py

ThreeDMatchConfig

utils/config.py

data/3DMatch

dataset/ThreeDMatch.py

The testing with the pretrained models on 3DMatch can by easily done by changing the path of log in

test_3dmatch.py

chosen_log = 'path_to_pretrained_log'

and runing

python test_3dmatch.py

The descriptors and detection scores for each point will be generated and saved in

geometric_registration/D3Feat_{timestr}/

Feature Matching Recall

inlier ratio

bash
cd geometric_registration/
python evaluate.py D3Feat [timestr of the model]

Registration Recall

evaluate.m

geometric_registration/3dmatch

descriptorName

D3Feat_{timestr}

geometric_registration/3dmatch/evaluate.m

evaluate.py

KITTI

Similarly, the training and testing of KITTI data set can be done by running

bash
python training_KITTI.py

bash
python test_KITTI.py

geometric_registration_kitti/D3Feat_{timestr}/

Relative Rotation Error

Relative Translation Error

utils/test.py

Keypoint Repeatability

After generating the descriptors and detection scores (which will be saved in

geometric_registration

geometric_registration_kitti

cd repeatability/
python evaluate_3dmatch_our.py D3Feat [timestr of the model]

or

cd repeatability/
python evaluate_kitti_our.py D3Feat [timestr of the model]

Pretrained Model

We provide the pre-trained model of 3DMatch in

results/

results_kitti/

Post-Conference Update

• Training Loss: We have found that circle loss provides an insightful idea for metric learning area and shows better and fast convergence for training D3Feat. To enable it, please change the losstype to `'circleloss'

  in

  KPFCNN_model.py

  , and the hyper-paramters for circle loss can be changed in

  loss.py`.

References

[1] 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions, Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser, CVPR 2017.

[2] The Perfect Match: 3D Point Cloud Matching with Smoothed Densities, Zan Gojcic, Caifa Zhou, Jan D. Wegner, and Andreas Wieser, CVPR 2019.

[3] Fully Convolutional Geometric Features: Christopher Choy and Jaesik Park and Vladlen Koltun, ICCV 2019.