Neural Point-Based Graphics

Project   Video   Paper

Neural Point-Based Graphics
Kara-Ali Aliev¹   Artem Sevastopolsky^1,2   Maria Kolos^1,2   Dmitry Ulyanov³   Victor Lempitsky^1,2
1Samsung AI Center   ¹Skolkovo Institute of Science and Technology   ³in3d.io

UPD (09.02.2021): added a Docker container which can be executed on a headless node. See Docker Readme.

About

This is PyTorch implementation of Neural Point-Based Graphics (NPBG), a new method for realtime photo-realistic rendering of real scenes. NPBG uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network.

Setup

The following instructions describe installation of conda environment. If you wish to setup the Docker environment, see the Readme in the docker folder. This way is also recommended for headless machines (without X server enabled).

Run this command to install python environment using conda:

bash
source scripts/install_deps.sh

Run

You can render one of the fitted scenes we provide right away in the real-time viewer or fit your own scene.

Download fitted scenes and universal rendering network weights from here and unpack in the sources root directory.

We suppose that you have at least one GeForce GTX 1080 Ti for fitting and inference.

Viewer navigation:

• Rotation: press left mouse button and drag
• Move: press rigth mouse button and drug / scroll middle mouse botton
• Pan: press middle mouse button and drug

Use fitted scene

Here we show a couple of examples how to run fitted scenes in the viewer.

Person 1

python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view

Since this scene was fitted on 4k images, we crop image size with

--viewport

Studio

python viewer.py --config downloads/studio.yaml --rmode fly

Check

downloads

Fit your scene

Fitting a new scene consists of two steps:

1. Point cloud reconstruction
2. Fitting descriptors

There is a bunch of software for point cloud reconstruction. While it is possible to adopt different software packages for our pipeline, we will choose Agisoft Metashape for this demonstration.

Point cloud reconstruction (Agisoft Metashape)

If you don't have a license for Agisoft Metashape Pro, start a trial version by filling in the form. On the first start, enter you license key.

Download and install Agisoft Metashape:

bash
wget http://download.agisoft.com/metashape-pro_1_6_2_amd64.tar.gz
tar xvf metashape-pro_1_6_2_amd64.tar.gz
cd metashape-pro
LD_LIBRARY_PATH="python/lib:$LD_LIBRARY_PATH" ./python/bin/python3.5 -m pip install pillow
bash metashape.sh

Optionally, enable GPU acceleration by checking Tools -> Preferences -> GPU.

Depending on the specs of your PC you may need to downscale your images to proper size. We recommend using 4k images or less. For example, if you want to downscale images by a factor of two, run this command: ```bash

convert comes with imagemagick package

sudo apt install imagemagick

in images directory

for fn in *jpg; do convert $fn -resize 50% $fn; done ```

Build point cloud:

bash
bash metashape.sh -r /scripts/metashape_build_cloud.py 

images

The script will produce: *

point_cloud.ply

cameras.xml

images_undistorted

project.psz

scene.yaml

Make sure the point cloud has no severe misalignments and crop out unnecessary geometry to optimize memory consumption. To edit a scene, open

project.psz

Now we can fit descriptors for this scene.

Fitting descriptors

Modify

configs/paths_example.yaml

Fit the scene:

bash
python train.py --config configs/train_example.yaml --pipeline npbg.pipelines.ogl.TexturePipeline --dataset_names 

paths_example.yaml

data/logs

The command above will finetune weights of the rendering network. This regime usually produces more appealing results. To freeze the rendering network, use option

--freeze_net

downloads/weights

net_ckpt

train_example.yaml

If you have masks for target images, use option '--use_masks'. Make sure masks align with target images.

When the model converge (usually 10 epochs is enough), run the scene in the viewer:

python viewer.py --config .yaml --checkpoint data/logs//checkpoints/.pth --origin-view

where

.yaml

--checkpoint

--origin-view

model3d_origin

.yaml

downloads/person_1.yaml

Guidelines for fitting novel scenes

Fitting novel scenes can sometimes be tricky, most often due to the preparation of camera poses that are provided in different ways by different sources, or sometimes because of the reconstruction issues (see below). We recommend checking out this and this issues for detailed explanations.

The most important insight is related to the configs structure. There is a system of 3 configs used in NPBG:

(there is another optional config -- inference config, which is essentially a scene config with

net_ckpt

texture_ckpt

To fit a new scene, one should a scene config

configs/my_scene_name.yaml

configs/my_scene_paths.yaml

Code logic and structure

Since our repository is based on a custom, specific framework, we leave the following diagram with the basic code logic. For those who wish to extend our code with additional features or try out related ideas (which we would highly appreciate), this diagram should help finding where the changes should be applied in the code. At the same time, various technical intricacies are not shown here for the sake of clarity.

Issues

• Reconstruction failure and geometry misalignment. Taking photos for photogrammetry is the most critical part in the whole pipeline. Follow these recommendations to have a good reconstruction:
  • Set the highest quality in the camera settings;
  • Keep the object in focus, don't set f-stop too low;
  • Fix shutter speed, the faster the better, but don't underexpose the photo;
  • Photos must be bright enough, but don't overexpose the photo;
  • Keep ISO low enough as it may introduce noise;
  • Keep the objects still, remove moving parts from the scene;
  • Take photos with at least 70% overlap;
  • If possible, use soft diffused light;
  • Avoid dark shadows;

If you are using a smarthone with Android, OpenCamera may come handy. A good starting point for settings is f/8, ISO 300, shutter speed 1/125s. iPhone users are recommended to fix exposure in the Camera. Follow this guide for more recommendations. * Viewer performance. If Pytorch and X server run on different GPUs there will be extra data transfer overhead between two GPUs. If higher framerate is desirable, make sure they run on the same GPU (use

CUDA_VISIBLE_DEVICES

CUDA_VISIBLE_DEVICES

TODOs

This is what we want to implement as well. We would also highly appreciate the help from the community.

• Point cloud reconstruction with COLMAP. As Agisoft Metashape is a proprietary software, the community would most benefit from an open source package like COLMAP which has almost the same functionality as Metashape, so the goal is to have

  colmap_build_cloud.py

  script working in the same manner as

  metashape_build_cloud.py

  .
• Convenient viewer navigation. Interactively choosing rotation center would make navigation much more conveniet. At the moment the viewer either explicitly imports the model's origin matrix or sets the origin automatically based on point cloud density.

Citation

@article{Аliev2020,
    title={Neural Point-Based Graphics},
    author={Kara-Ali Aliev and Artem Sevastopolsky and Maria Kolos and Dmitry Ulyanov and Victor Lempitsky},
    year={2020},
    eprint={1906.08240v3},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}