Need help with learning3d?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

vinits5
257 Stars 42 Forks MIT License 76 Commits 3 Opened issues

Description

This is a complete package of recent deep learning methods for 3D point clouds in pytorch (with pretrained models).

Services available

!
?

Need anything else?

Contributors list

No Data

Learning3D: A Modern Library for Deep Learning on 3D Point Clouds Data.

Documentation | Blog | Demo

Learning3D is an open-source library that supports the development of deep learning algorithms that deal with 3D data. The Learning3D exposes a set of state of art deep neural networks in python. A modular code has been provided for further development. We welcome contributions from the open-source community.

Latest News:

  1. [24 Dec. 2020]: MaskNet is now ready to enhance the performance of registration algorithms in learning3d for occluded point clouds.
  2. [24 Dec. 2020]: Loss based on the predicted and ground truth correspondences is added in learning3d after consideration of Correspondence Matrices are Underrated paper.
  3. [24 Dec. 2020]: PointConv, latent feature estimation using convolutions on point clouds is now available in learning3d.
  4. [16 Oct. 2020]: DeepGMR, registration using gaussian mixture models is now available in learning3d
  5. [14 Oct. 2020]: Now, use your own data in learning3d. (Check out UserData functionality!)

Available Computer Vision Algorithms in Learning3D

| Sr. No. | Tasks | Algorithms | |:-------------:|:----------:|:-----| | 1 | Classification | PointNet, DGCNN, PPFNet, PointConv | | 2 | Segmentation | PointNet, DGCNN | | 3 | Reconstruction | Point Completion Network (PCN) | | 4 | Registration | PointNetLK, PCRNet, DCP, PRNet, RPM-Net, DeepGMR | | 5 | Flow Estimation | FlowNet3D | | 6 | Inlier Estimation | MaskNet |

Available Pretrained Models

  1. PointNet
  2. PCN
  3. PointNetLK
  4. PCRNet
  5. DCP
  6. PRNet
  7. FlowNet3D
  8. RPM-Net (clean-trained.pth, noisy-trained.pth, partial-pretrained.pth)
  9. DeepGMR
  10. PointConv (Download from this link)
  11. MaskNet

Available Datasets

  1. ModelNet40

Available Loss Functions

  1. Classification Loss (Cross Entropy)
  2. Registration Losses (FrobeniusNormLoss, RMSEFeaturesLoss)
  3. Distance Losses (Chamfer Distance, Earth Mover's Distance)
  4. Correspondence Loss (based on this paper)

Technical Details

Supported OS

  1. Ubuntu 16.04
  2. Ubuntu 18.04
  3. Linux Mint

Requirements

  1. CUDA 10.0 or higher
  2. Pytorch 1.3 or higher

How to use this library?

Important Note: Clone this repository in your project. Please don't add your codes in "learning3d" folder.

  1. All networks are defined in the module "models".
  2. All loss functions are defined in the module "losses".
  3. Data loaders are pre-defined in data_utils/dataloaders.py file.
  4. All pretrained models are provided in learning3d/pretrained folder.

Documentation

B: Batch Size, N: No. of points and C: Channels.

Use of Point Embedding Networks:

from learning3d.models import PointNet, DGCNN, PPFNet\ pn = PointNet(embdims=1024, inputshape='bnc', usebn=False)\ dgcnn = DGCNN(embdims=1024, inputshape='bnc')\ ppf = PPFNet(features=['ppf', 'dxyz', 'xyz'], embdims=96, radius='0.3', num_neighbours=64)

| Sr. No. | Variable | Data type | Shape | Choices | Use | |:---:|:---:|:---:|:---:|:---:|:---:| | 1. | embdims | Integer | Scalar | 1024, 512 | Size of feature vector for the each point| | 2. | inputshape | String | - | 'bnc', 'bcn' | Shape of input point cloud| | 3. | output | tensor | BxCxN | - | High dimensional embeddings for each point| | 4. | features | List of Strings | - | ['ppf', 'dxyz', 'xyz'] | Use of various features | | 5. | radius | Float | Scalar | 0.3 | Radius of cluster for computing local features | | 6. | num_neighbours | Integer | Scalar | 64 | Maximum number of points to consider per cluster |

Use of Classification / Segmentation Network:

from learning3d.models import Classifier, PointNet, Segmentation\ classifier = Classifier(featuremodel=PointNet(), numclasses=40)\ seg = Segmentation(featuremodel=PointNet(), numclasses=40)

| Sr. No. | Variable | Data type | Shape | Choices | Use | |:---:|:---:|:---:|:---:|:---:|:---:| | 1. | featuremodel | Object | - | PointNet / DGCNN | Point cloud embedding network | | 2. | numclasses | Integer | Scalar | 10, 40 | Number of object categories to be classified | | 3. | output | tensor | Classification: Bx40, Segmentation: BxNx40 | 10, 40 | Probabilities of each category or each point |

Use of Registration Networks:

from learning3d.models import PointNet, PointNetLK, DCP, iPCRNet, PRNet, PPFNet, RPMNet\ pnlk = PointNetLK(featuremodel=PointNet(), delta=1e-02, xtol=1e-07, p0zeromean=True, p1zeromean=True, pooling='max')\ dcp = DCP(featuremodel=PointNet(), pointer='transformer', head='svd')\ pcrnet = iPCRNet(featuremoodel=PointNet(), pooling='max')\ rpmnet = RPMNet(featuremodel=PPFNet())\ deepgmr = DeepGMR(userri=True, featuremodel=PointNet(), nearestneighbors=20)

| Sr. No. | Variable | Data type | Choices | Use | Algorithm | |:---:|:---:|:---:|:---:|:---:|:---:| | 1. | featuremodel | Object | PointNet / DGCNN | Point cloud embedding network | PointNetLK | | 2. | delta | Float | Scalar | Parameter to calculate approximate jacobian | PointNetLK | | 3. | xtol | Float | Scalar | Check tolerance to stop iterations | PointNetLK | | 4. | p0zeromean | Boolean | True/False | Subtract mean from template point cloud | PointNetLK | | 5. | p1zeromean | Boolean | True/False | Subtract mean from source point cloud | PointNetLK | | 6. | pooling | String | 'max' / 'avg' | Type of pooling used to get global feature vectror | PointNetLK | | 7. | pointer | String | 'transformer' / 'identity' | Choice for Transformer/Attention network | DCP | | 8. | head | String | 'svd' / 'mlp' | Choice of module to estimate registration params | DCP | | 9. | userri | Boolean | True/False | Use nearest neighbors to estimate point cloud features. | DeepGMR | | 10. | nearestneighbores | Integer | 20/any integer | Give number of nearest neighbors used to estimate features | DeepGMR |

Use of Inlier Estimation Network (MaskNet):

from learning3d.models import MaskNet, PointNet\ masknet = MaskNet(featuremodel=PointNet(), istraining=True)

| Sr. No. | Variable | Data type | Choices | Use | |:---:|:---:|:---:|:---:|:---:| | 1. | featuremodel | Object | PointNet / DGCNN | Point cloud embedding network | | 2. | istraining | Boolean | True / False | Specify if the network will undergo training or testing |

Use of Point Completion Network:

from learning3d.models import PCN\ pcn = PCN(embdims=1024, inputshape='bnc', numcoarse=1024, gridsize=4, detailed_output=True)

| Sr. No. | Variable | Data type | Choices | Use | |:---:|:---:|:---:|:---:|:---:| | 1. | embdims | Integer | 1024, 512 | Size of feature vector for each point | | 2. | inputshape | String | 'bnc' / 'bcn' | Shape of input point cloud | | 3. | numcoarse | Integer | 1024 | Shape of output point cloud | | 4. | gridsize | Integer | 4, 8, 16 | Size of grid used to produce detailed output | | 5. | detailed_output | Boolean | True / False | Choice for additional module to create detailed output point cloud|

Use of PointConv:

Use the following to create pretrained model provided by authors.

from learning3d.models import createpointconv\ PointConv = createpointconv(classifier=True, pretrained='path of checkpoint')\ ptconv = PointConv(embdims=1024, inputshape='bnc', inputchanneldim=6, classifier=True)

OR\ Use the following to create your own PointConv model.

PointConv = createpointconv(classifier=False, pretrained=None)\ ptconv = PointConv(embdims=1024, inputshape='bnc', inputchannel_dim=3, classifier=True)

PointConv variable is a class. Users can use it to create a sub-class to override create_classifier and create_structure methods in order to change PointConv's network architecture.

| Sr. No. | Variable | Data type | Choices | Use | |:---:|:---:|:---:|:---:|:---:| | 1. | embdims | Integer | 1024, 512 | Size of feature vector for each point | | 2. | inputshape | String | 'bnc' / 'bcn' | Shape of input point cloud | | 3. | inputchanneldim | Integer | 3/6 | Define if point cloud contains only xyz co-ordinates or normals and colors as well | | 4. | classifier | Boolean | True / False | Choose if you want to use a classifier with PointConv | | 5. | pretrained | Boolean | String | Give path of the pretrained classifier model (only use it for weights given by authors) |

Use of Flow Estimation Network:

from learning3d.models import FlowNet3D\ flownet = FlowNet3D()

Use of Data Loaders:

from learning3d.datautils import ModelNet40Data, ClassificationData, RegistrationData, FlowData\ modelnet40 = ModelNet40Data(train=True, numpoints=1024, download=True)\ classificationdata = ClassificationData(dataclass=ModelNet40Data())\ registrationdata = RegistrationData(algorithm='PointNetLK', dataclass=ModelNet40Data(), partialsource=False, partialtemplate=False, noise=False)\ flow_data = FlowData()

| Sr. No. | Variable | Data type | Choices | Use | |:---:|:---:|:---:|:---:|:---:| | 1. | train | Boolean | True / False | Split data as train/test set | | 2. | numpoints | Integer | 1024 | Number of points in each point cloud | | 3. | download | Boolean | True / False | If data not available then download it | | 4. | dataclass | Object | - | Specify which dataset to use | | 5. | algorithm | String | 'PointNetLK', 'PCRNet', 'DCP', 'iPCRNet' | Algorithm used for registration | | 6. | partialsource | Boolean | True / False | Create partial source point cloud | | 7. | partialtemplate | Boolean | True / False | Create partial template point cloud | | 8. | noise | Boolean | True / False | Add noise in source point cloud |

Use Your Own Data:

from learning3d.datautils import UserData\ dataset = UserData(application, datadict)

|Sr. No. | Application | Required Key | Respective Value | |:---:|:---:|:---:|:---:| | 1. | 'classification' | 'pcs' | Point Clouds (BxNx3) | | | | 'labels' | Ground Truth Class Labels (BxN) | | 2. | 'registration' | 'template' | Template Point Clouds (BxNx3) | | | | 'source' | Source Point Clouds (BxNx3) | | | | 'transformation' | Ground Truth Transformation (Bx4x4)| | 3. | 'flow_estimation' | 'frame1' | Point Clouds (BxNx3) | | | | 'frame2' | Point Clouds (BxNx3) | | | | 'flow' | Ground Truth Flow Vector (BxNx3)|

Use of Loss Functions:

from learning3d.losses import RMSEFeaturesLoss, FrobeniusNormLoss, ClassificationLoss, EMDLoss, ChamferDistanceLoss, CorrespondenceLoss\ rmse = RMSEFeaturesLoss()\ fnloss = FrobeniusNormLoss()\ classificationloss = ClassificationLoss()\ emd = EMDLoss()\ cd = ChamferDistanceLoss()\ corr = CorrespondenceLoss()

| Sr. No. | Loss Type | Use | |:---:|:---:|:---:| | 1. | RMSEFeaturesLoss | Used to find root mean square value between two global feature vectors of point clouds | | 2. | FrobeniusNormLoss | Used to find frobenius norm between two transfromation matrices | | 3. | ClassificationLoss | Used to calculate cross-entropy loss | | 4. | EMDLoss | Earth Mover's distance between two given point clouds | | 5. | ChamferDistanceLoss | Chamfer's distance between two given point clouds | | 6. | CorrespondenceLoss | Computes cross entropy loss using the predicted correspondence and ground truth correspondence for each source point |

To run codes from examples:

  1. Copy the file from "examples" folder outside of the directory "learning3d"
  2. Now, run the file. (ex. python test_pointnet.py)
  3. Your Directory/Location
    • learning3d
    • test_pointnet.py

References:

  1. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  2. Dynamic Graph CNN for Learning on Point Clouds
  3. PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
  4. PointConv: Deep Convolutional Networks on 3D Point Clouds
  5. PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
  6. PCRNet: Point Cloud Registration Network using PointNet Encoding
  7. Deep Closest Point: Learning Representations for Point Cloud Registration
  8. PRNet: Self-Supervised Learning for Partial-to-Partial Registration
  9. FlowNet3D: Learning Scene Flow in 3D Point Clouds
  10. PCN: Point Completion Network
  11. RPM-Net: Robust Point Matching using Learned Features
  12. 3D ShapeNets: A Deep Representation for Volumetric Shapes
  13. DeepGMR: Learning Latent Gaussian Mixture Models for Registration
  14. CMU: Correspondence Matrices are Underrated
  15. MaskNet: A Fully-Convolutional Network to Estimate Inlier Points

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.