Implementation of CVPR'20 oral paper - D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features https://arxiv.org/abs/2003.03164
No Data
TensorFlow implementation of D3Feat for CVPR'2020 Oral paper "D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features", by Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan and Chiew-Lan Tai.
This paper focus on dense feature detection and description for 3D point clouds in a joint manner. If you find this project useful, please cite:
@article{bai2020d3feat, title={D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features}, author={Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan and Chiew-Lan Tai}, journal={arXiv:2003.03164 [cs.CV]}, year={2020} }
The PyTorch implementation can be found here.
A successful point cloud registration often lies on robust establishment of sparse matches through discriminative 3D local features. Despite the fast evolution of learning-based 3D feature descriptors, little attention has been drawn to the learning of 3D feature detectors, even less for a joint learning of the two tasks. In this paper, we leverage a 3D fully convolutional network for 3D point clouds, and propose a novel and practical learning mechanism that densely predicts both a detection score and a description feature for each 3D point. In particular, we propose a keypoint selection strategy that overcomes the inherent density variations of 3D point clouds, and further propose a self-supervised detector loss guided by the on-the-fly feature matching results during training. Finally, our method achieves state-of-the-art results in both indoor and outdoor scenarios, evaluated on 3DMatch and KITTI datasets, and shows its strong generalization ability on the ETH dataset. Towards practical use, we show that by adopting a reliable feature detector, sampling a smaller number of features is sufficient to achieve accurate and fast point cloud alignment.
Create the environment and install the required libaries:
conda env create -f environment.yml
Compile the customized Tensorflow operators located in
tf_custom_ops. Open a terminal in this folder, and run:
sh compile_op.sh
Compile the C++ extension module for python located in
cpp_wrappers. Open a terminal in this folder, and run:
sh compile_wrappers.sh
The code is heavily borrowed from KPConv. You can find the guidance for compiling the tensorflow operators and C++ wrappers in INSTALL.md.
We provide a small demo to extract dense feature and detection score for two point cloud, and register them using RANSAC. The ply files are saved in the
demo_datafolder, which can be replaced by your own data. Now we are using two point cloud fragments from 3DMatch dataset. To try the demo, please run
bash python demo_registration.pyIt will compute the descriptors and detection scores using the released weight on 3DMatch dataset, and save them in .npz file in
demo_data. These descriptors are then used to estimate the rigid-body transformation parameters using RANSAC. Visualization of the inital state and registered state will show up.
We also visualize the detected keypoints on two point cloud.
3DMatch
The training set of 3DMatch[1] can be downloaded from here. It is generated by
datasets/cal_overlap.pywhich select all the point cloud fragments pairs having more than 30% overlap.
The test set point clouds and the evaluation files(for registration recall) can be downloaded from 3DMatch Geometric Registration website.
Please put the training set under
data/3DMatchfolder and the test set under
data/3DMatch/fragments. And I have already put the ground truth poses in the
geometric_registration/gt_resultfolder.
KITTI
The training and test set can be download from KITTI Odometry website. I follow the FCGF[3] for pre-processing.
ETH
The test set (we only use ETH dataset to evaluate the generalization ability of our method) can be downloaded from here. Detail instructions can be found in PerfectMatch[2].
The training on 3DMatch dataset can be done by running
bash python training_3dmatch.pyThis file contains a configuration subclass
ThreeDMatchConfig, inherited from the general configuration class Config defined in
utils/config.py. The value of every parameter can be modified in the subclass. The default path to 3DMatch training set is
data/3DMatch, which can be changed in
dataset/ThreeDMatch.py.
The testing with the pretrained models on 3DMatch can by easily done by changing the path of log in
test_3dmatch.pyfile
chosen_log = 'path_to_pretrained_log'
and runing
python test_3dmatch.py
The descriptors and detection scores for each point will be generated and saved in
geometric_registration/D3Feat_{timestr}/folder. Then the
Feature Matching Recalland
inlier ratiocan be caluclated by running
bash cd geometric_registration/ python evaluate.py D3Feat [timestr of the model]The
Registration Recallcan be calculated by running the
evaluate.min
geometric_registration/3dmatchwhich are provided by 3DMatch. You need to modify the
descriptorNameto
D3Feat_{timestr}in the
geometric_registration/3dmatch/evaluate.mfile. You can change the number of keypoints in
evaluate.py
Similarly, the training and testing of KITTI data set can be done by running
bash python training_KITTI.pyAnd
bash python test_KITTI.pyThe detected keypoints and scores for each fragment, as well as the estimated transformation matrix between each ground truth pair will be saved in
geometric_registration_kitti/D3Feat_{timestr}/folder. Then the
Relative Rotation Errorand
Relative Translation Errorwill be calculated by comparing the ground truth pose and estimated pose. The code of this part is heavily borrowed from FCGF[3]. You can change the number of keypoints in
utils/test.py.
After generating the descriptors and detection scores (which will be saved in
geometric_registrationor
geometric_registration_kitti), the keypoint repeatbility can be calculated by running
cd repeatability/ python evaluate_3dmatch_our.py D3Feat [timestr of the model]
or
cd repeatability/ python evaluate_kitti_our.py D3Feat [timestr of the model]
We provide the pre-trained model of 3DMatch in
results/and KITTI in
results_kitti/.
inKPFCNN_model.py
, and the hyper-paramters for circle loss can be changed inloss.py`.
[1] 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions, Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser, CVPR 2017.
[2] The Perfect Match: 3D Point Cloud Matching with Smoothed Densities, Zan Gojcic, Caifa Zhou, Jan D. Wegner, and Andreas Wieser, CVPR 2019.
[3] Fully Convolutional Geometric Features: Christopher Choy and Jaesik Park and Vladlen Koltun, ICCV 2019.