Virtual Video Device for Background Replacement with Deep Semantic Segmentation
In these modern times where everyone is sitting at home and skype-ing/zoom-ing/webrtc-ing all the time, I was a bit annoyed about always showing my messy home office to the world. Skype has a "blur background" feature, but that starts to get boring after a while (and it's less private than I would personally like). Zoom has some background substitution thingy built-in, but I'm not touching that software with a bargepole (and that feature is not available on Linux anyway). So I decided to look into how to roll my own implementation without being dependent on any particular video conferencing software to support this.
This whole shebang involves three main steps with varying difficulty: - find person in video (hard) - replace background (easy) - pipe data to virtual video device (medium)
I've been working a lot with depth cameras previously, also for background segmentation (see SurfaceStreams), so I just grabbed a leftover RealSense camera from the lab and gave it a shot. However, the depth data in a cluttered office environment is quite noisy, and no matter how I tweaked the camera settings, it could not produce any depth data for my hair...? I looked like a medieval monk who had the top of his head chopped off, so ... next.
See https://docs.opencv.org/3.4/d1/dc5/tutorialbackgroundsubtraction.html for tutorial. Should work OK for mostly static backgrounds and small moving objects, but does not work for a mostly static person in front of a static background. Next.
See https://docs.opencv.org/3.4/db/d28/tutorialcascadeclassifier.html for tutorial. Works okay-ish, but obviously only detects the face, and not the rest of the person. Also, only roughly matches an ellipse which is looking rater weird in the end. Next.
I've heard good things about this deep learning stuff, so let's try that. I first had to find my way through a pile of frameworks (Keras, Tensorflow, PyTorch, etc.), but after I found a ready-made model for semantic segmentation based on Tensorflow Lite (DeepLab v3+), I settled on that.
I had a look at the corresponding Python example, C++ example, and Android example, and based on those, I first cobbled together a Python demo. That was running at about 2.5 FPS, which is really excruciatingly slow, so I built a C++ version which manages 10 FPS without too much hand optimization. Good enough.
This is basically one line of code with OpenCV:
bg.copyTo(raw,mask);Told you that's the easy part.
I'm using v4l2loopback to pipe the data from my userspace tool into any software that can open a V4L2 device. This isn't too hard because of the nice examples, but there are some catches, most notably color space. It took quite some trial and error to find a common pixel format that's accepted by Firefox, Skype, and guvcview, and that is YUYV. Nicely enough, my webcam can output YUYV directly as raw data, so that does save me some colorspace conversions.
The dataflow through the whole program is roughly as follows:
write()data to virtual video device
(*) these are required input parameters for DeepLab v3+
Tested with the following dependencies: - Ubuntu 18.04.5, x86-64 - Linux kernel 4.15 (stock package) - OpenCV 3.2.0 (stock package) - V4L2-Loopback 0.10.0 (stock package) - Tensorflow Lite 2.1.0 (from repo) - Ultra-short build guide for Tensorflow Lite C++ library: clone repo above, then... - run
Tested with the following software: - Firefox - 74.0.1 (works) - 76.0.1 (works) - Skype - 18.104.22.168 (works) - 22.214.171.124 (works) - guvcview 2.0.5 (works with parameter
-c read) - Microsoft Teams 1.3.00.5153 (works) - Chrome 81.0.4044.138 (works) - Zoom 5.0.403652.0509 (works - yes, I'm a hypocrite, I tested it with Zoom after all :-)
First, load the v4l2loopback module (extra settings needed to make Chrome work):
sudo modprobe v4l2loopback devices=1 max_buffers=2 exclusive_caps=1 card_label="VirtualCam"Then, run deepseg (-d for debug, -c for capture device, -v for virtual device):
./deepseg -d -c /dev/video0 -v /dev/video1
As usual: pull requests welcome. - The project name isn't catchy enough. Help me find a nice backronym. - Resolution is currently hardcoded to 640x480 (lowest common denominator). - Only works with Linux, because that's what I use. - Needs a webcam that can produce raw YUYV data (but extending to the common YUV420 format should be trivial) - CPU hog: maxes out two cores on my 2.7 GHz i5 machine for just VGA @ 10 FPS. - Uses stock Deeplab v3+ network. Maybe re-training with only "person" and "background" classes could improve performance?
Firefox preferred formats: https://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/videocapture/linux/videocapture_linux.cc#142-159