Robust realtime face and facial landmark tracking on CPU with Unity integration
Note: This is a tracking library, not a stand-alone avatar puppeteering program. I'm also working on such an application, which is currently in a public beta.
This project implements a facial landmark detection model based on MobileNetV3.
As Pytorch 1.3 CPU inference speed on Windows is very low, the model was converted to ONNX format. Using onnxruntime it can run at 30 - 60 fps tracking a single face. There are four models, with different speed to tracking quality trade-offs.
If anyone is curious, the name is a silly pun on the open seas and seeing faces. There's no deeper meaning.
An up to date sample video can be found here, showing the default tracking model's performance under different noise and light levels.
A sample Unity project for VRM based avatar animation can be found here.
The face tracking itself is done by the
facetracker.pyPython 3.7 script. It is a commandline program, so you should start it manually from cmd or write a batch file to start it. If you downloaded a release and are on Windows, you can run the
facetracker.exeinside the
Binaryfolder without having Python installed. You can also use the
run.batinside the
Binaryfolder for a basic demonstration of the tracker.
The script will perform the tracking on webcam input or video file and send the tracking data over UDP. This design also allows tracking to be done on a separate PC from the one who uses the tracking information. This can be useful to enhance performance and to avoid accidentially revealing camera footage.
The provided
OpenSeeUnity component can receive these UDP packets and provides the received information through a public field called
trackingData. The
OpenSeeShowPointscomponent can visualize the landmark points of a detected face. It also serves as an example. Please look at it to see how to properly make use of the
OpenSeecomponent. Further examples are included in the
Examplesfolder. The UDP packets are received in a separate thread, so any components using the
trackingDatafield of the
OpenSeecomponent should first copy the field and access this copy, because otherwise the information may get overwritten during processing. This design also means that the field will keep updating, even if the
OpenSeecomponent is disabled.
Run the python script with
--helpto learn about the possible options you can set.
python facetracker.py --help
A simple demonstration can be achieved by creating a new scene in Unity, adding an empty game object and both the
OpenSeeand
OpenSeeShowPointscomponents to it. While the scene is playing, run the face tracker on a video file:
python facetracker.py --visualize 3 --pnp-points 1 --max-threads 4 -c video.mp4
This way the tracking script will output its own tracking visualization while also demonstrating the transmission of tracking data to Unity.
The included
OpenSeeLaunchercomponent allows starting the face tracker program from Unity. It is designed to work with the pyinstaller created executable distributed in the binary release bundles. It provides three public API functions:
public string[] ListCameras()returns the names of available cameras. The index of the camera in the array corresponds to its ID for the
cameraIndexfield. Setting the
cameraIndexto
-1will disable webcam capturing.
public bool StartTracker()will start the tracker. If it is already running, it will shut down the running instance and start a new one with the current settings.
public void StopTracker()will stop the tracker. The tracker is stopped automatically when the application is terminated or the
OpenSeeLauncherobject is destroyed.
The
OpenSeeLaunchercomponent uses WinAPI job objects to ensure that the tracker child process is terminated if the application crashes or closes without terminating the tracker process first.
Additional custom commandline arguments should be added one by one into elements of
commandlineArgumentsarray. For example
-v 1should be added as two elements, one element containing
-vand one containing
1, not a single one containing both parts.
The included
OpenSeeIKTargetcomponent can be used in conjunction with FinalIK or other IK solutions to animate head motion.
The
OpenSeeExpressioncomponent can be added to the same component as the
OpenSeeFacecomponent to detect specific facial expressions. It has to be calibrated on a per-user basis. It can be controlled either through the checkboxes in the Unity Editor or through the equivalent public methods that can be found in its source code.
To calibrate this system, you have to gather example data for each expression. If the capture process is going too fast, you can use the
recordingSkipoption to slow it down.
The general process is as follows:
To delete the captured data for an expression, type in its name and tick the "Clear" box.
To save both the trained model and the captured training data, type in a filename including its full path in the "Filename" field and tick the "Save" box. To load it, enter the filename and tick the "Load" box.
--model 3, the fastest model with the lowest tracking quality is
--model 0.
--scan-everyframes. This can slow things down, so try to set
--facesno higher than the actual number of faces you are tracking.
Four pretrained face landmark models are included. Using the
--modelswitch, it is possible to select them for tracking. The given fps values are for running the model on a single face video on a single CPU core. Lowering the frame rate would reduce CPU usage by a corresponding degree.
FPS measurements are from running on one core of my CPU.
Pytorch weights for use with
model.pycan be found here.
More samples: Results3.png, Results4.png
The landmark model is quite robust with respect to the size and orientation of the faces, so the custom face detection model gets away with rougher bounding boxes than other approaches. It has a favorable speed to accuracy ratio for the purposes of this project.
The builds in the release section of this repository contain a
facetracker.exeinside a
Binaryfolder that was built using
pyinstallerand contains all required dependencies.
To run it, at least the
modelsfolder has to be placed in the same folder as
facetracker.exe. Placing it in a common parent folder should work too.
When distributing it, you should also distribute the
Licensesfolder along with it to make sure you conform to requirements set forth by some of the third party libraries. Unused models can be removed from redistributed packages without issue.
The release builds contain a custom build of ONNX Runtime without telemetry.
The required libraries can be installed using pip:
pip install onnxruntime opencv-python pillow numpy
The model was trained on a 66 point version of the LS3D-W dataset.
@inproceedings{bulat2017far, title={How far are we from solving the 2D \& 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)}, author={Bulat, Adrian and Tzimiropoulos, Georgios}, booktitle={International Conference on Computer Vision}, year={2017} }
Additional training has been done on the WFLW dataset after reducing it to 66 points and replacing the contour points and tip of the nose with points predicted by the model trained up to this point. This additional training is done to improve fitting to eyes and eyebrows.
@inproceedings{wayne2018lab, author = {Wu, Wayne and Qian, Chen and Yang, Shuo and Wang, Quan and Cai, Yici and Zhou, Qiang}, title = {Look at Boundary: A Boundary-Aware Face Alignment Algorithm}, booktitle = {CVPR}, month = June, year = {2018} }
For the training the gaze and blink detection model, the MPIIGaze dataset was used. Additionally, around 125000 synthetic eyes generated with UnityEyes were used during training.
The heatmap regression based face detection model was trained on random 224x224 crops from the WIDER FACE dataset.
@inproceedings{yang2016wider, Author = {Yang, Shuo and Luo, Ping and Loy, Chen Change and Tang, Xiaoou}, Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, Title = {WIDER FACE: A Face Detection Benchmark}, Year = {2016} }
The algorithm is inspired by:
The MobileNetV3 code was taken from here.
For all training a modified version of Adaptive Wing Loss was used.
For expression detection, LIBSVM is used.
Face detection is done using a custom heatmap regression based face detection model or RetinaFace.
@inproceedings{deng2019retinaface, title={RetinaFace: Single-stage Dense Face Localisation in the Wild}, author={Deng, Jiankang and Guo, Jia and Yuxiang, Zhou and Jinke Yu and Irene Kotsia and Zafeiriou, Stefanos}, booktitle={arxiv}, year={2019} }
RetinaFace detection is based on this implementation. The pretrained model was modified to remove unnecessary landmark detection and converted to ONNX format for a resolution of 640x640.
Many thanks to everyone who helped me test things!
The code and models are distributed under the BSD 2-clause license.
You can find licenses of third party libraries used for binary builds in the
Licensesfolder.