This repository contains all the code to reproduce our results from our recent publication: - Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Lüthi, Sandro Schönborn and Thomas Vetter "Morphable Face Models - An Open Framework" ( edoc-unibas / IEEE ) IN: 13th IEEE Conference on Automatic Face and Gesture Recognition (FG 2018) pp. 75-82
After the following information, we list all nescessary steps that you need to take to aquire and prepare the data as well as to run the pipeline and the experiments in the next sections.
When you expericence some problems under Windows while importing the data, please use the branch updateScalismoFaces.
When you experience problems, you have questions or feedback please use the mailing list for Morphable face models - an open framework.
For the registration pipeline and the experiments to work properly, some data, such as reference templates and landmarks are needed. The files are available for download at Registration Pipeline Data. The download contains the following in detail:
You can copy the content of the zip folder into
pipeline-data. The coarse structure looks the following:
pipeline-data ├── data │ ├── incoming │ ├── bu3dfe │ │ ├── original ├── recognition-experiment
If needed, you can change the location of the
pipeline-datadirectory in the BU3DDataProvider.scala file.
To register the BU-3DFE you have to acquire the dataset here:
and copy the
We assume that you have sbt already installed. If not, please follow the instructions given here.
Generally you can run the code using SBT. An example is how to run it in the terminal with:
cd /code-directory/ sbt run
If you do not have enough memory use:
sbt -J-Xmx50g run
Then the different steps are then listed and can be executed by entering the number of the script or by using:
sbt "run-main package.Classname"
During the pipeline we do not use the BU3DFE database data directly but first convert the data to match our formats. This step is done only once as a pre-processing and the output can be reused whenever you run a new registration.
To convert the original data from the BU3DFE database to our format use the command:
sbt "run-main preprocessing.ConvertBu3DRawData"
Explain raw data preprocessing steps in script. (The script might need some cleanup.)
Pre-computing the neutral prior model can take quite some time. However, it has to be computed only once offline and is stored in
You can run the building process with:
sbt "run-main registration.BuildNeutralPrior"
The core expression model augments the neutral model with expression deformations.
sbt "run-main registration.BuildCoreExpressionModel"
This step is used to transform the reference landmarks to the new mean of the generated models and to change the uncertainty of the individual landmarks.
sbt "run-main preprocessing.PrepareReferenceLandmarks"
sbt -J-Xmx40g "run-main registration.Registration"
The model building contains two steps:
This process may need some time and memory. Once the first step, the color extraction is computed it can be reused if you change for example the mask of the model that should be built. But to change this you have to out comment the corresponding line in the source code.
sbt -mem 40000 "run-main modelbuilding.ModelBuilding"
First you have to download the Multi-PIE database and copy the necessary files to the correct folders. This is described in the README file in the folder recogniton-experiment (comes with seperate download of the Basel Face Pipeline Data). For those experiments you need the Basel Face Model 2009 and 2017, which can be downloaded at: Probabilistic Morphable Models
To run the 3D reconstructions from the Multi-PIE database, you may want to execute it multiple times in parallel since a single fit taks ~20 minutes:
sbt -mem 5000 "fitting.experiments.RecognitionMultiPiePose"And to calculate the recognition scores execute:
sbt -mem 5000 "fitting.experiments.RecognitionEvaluation"Those where the neutral scores. To perform the expression experiments, run:
sbt -mem 5000 "fitting.experiments.RecognitionMultiPieExpression" sbt -mem 5000 "fitting.experiments.RecognitionEvaluationEx"