Need help with LISRD?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

rpautrat
162 Stars 9 Forks MIT License 11 Commits 3 Opened issues

Description

Local Invariance Selection at Runtime for Descriptors (LISRD)

Services available

!
?

Need anything else?

Contributors list

# 137,437
Jupyter...
Python
Shell
11 commits

LISRD - Local Invariance Selection at Runtime for Descriptors

This repository contains the implementation of the paper: Online Invariance Selection for Local Feature Descriptors, R. Pautrat, V. Larsson, M. Oswald and M. Pollefeys (Oral at ECCV 2020).

LISRD offers the possibility to leverage descriptors with different invariances (e.g. invariant or variant to rotation and/or illumination) and to perform an online selection of the most adapted invariance when matching two images.

The following shows for example matches predicted with LISRD-SIFT that selects descriptors between SIFT (rotation invariant) and Upright SIFT (rotation variant): demo_lisrd_sift

Usage

Installation

Clone the repository with its submodule:

bash
git clone --recurse-submodules https://github.com/rpautrat/LISRD.git

We recommend using this code in a Python environment (e.g. venv or conda). The following script installs the necessary requirements and install the repository as a Python package locally:

bash
make install

Training your own model

All the training parameters should be present in a configuration file located in the folder

lisrd/configs
. The generic script to train your own model from the root folder is the following:
bash
python -m lisrd.experiment train  

For example, training LISRD with the 4 types of invariance used in the paper would be:

bash
python -m lisrd.experiment train lisrd/configs/lisrd.yaml ~/Documents/experiments/My_experiment
Use the config file
lisrd/configs/lisrd_sift.yaml
to instead train LISRD-SIFT that chooses between SIFT and Upright SIFT descriptors.

Pretrained models

We provide two pretrained models: - lisrd_aachen: version of LISRD used in the paper, trained on the MS COCO dataset, Multi-Illumination Images in the Wild and Aachen dataset images augmented with style-transfer or with optical flow information. - lisrd_vidit: new version of LISRD, trained on the MS COCO dataset, Multi-Illumination Images in the Wild and the Virtual Image Dataset for Illumination Transfer (VIDIT). The results below are obtained using this model.

How to use it

We provide a notebook showing how to use the trained models of LISRD. Additionally,

lisrd/export_features.yaml
is a script to export the LISRD descriptors with either SIFT or SuperPoint keypoints on a given set of images. It can be used as follows:
bash
python -m lisrd.export_features   --checkpoint  --keypoints  --num_kp 

Results on the RDNIM dataset

The Rotated Day-Night Image Matching (RDNIM) dataset originates from the DNIM dataset and has been augmented with homographic warps with 50% of the images including rotations. The images used for evaluation in the paper are available here.

Comparison to the state of the art on the RDNIM dataset, using SuperPoint keypoints for all methods and a correctness threshold of 3 pixels:

Day reference Night reference
Homography estimation Precision Recall Homography estimation Precision Recall
Root SIFT 0.134 0.184 0.125 0.186 0.239 0.182
HardNet 0.249 0.225 0.224 0.325 0.359 0.365
SOSNet 0.226 0.218 0.226 0.252 0.288 0.296
SuperPoint 0.178 0.191 0.214 0.235 0.259 0.296
D2-Net 0.124 0.196 0.145 0.195 0.265 0.218
R2D2 0.190 0.173 0.180 0.229 0.237 0.237
GIFT 0.225 0.155 0.149 0.294 0.240 0.229
LISRD (ours) 0.318 0.406 0.439 0.391 0.488 0.520

mma_rdnim

Bibtex

If you use this code in your project, please consider citing the following paper:

bibtex
@InProceedings{Pautrat_2020_ECCV,
    author = {Pautrat, Rémi and Larsson, Viktor and Oswald, Martin R. and Pollefeys, Marc},
    title = {Online Invariance Selection for Local Feature Descriptors},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
    year = {2020},
}

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.