Upconverting YouTube soccer videos in 3D for viewing in AR/VR devices.Soccer On Your Tabletop
Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, Steve Seitz, in CVPR 2018
Warning: I am in the process of transferring the repo, so some things probably will not work.
This repository contains:
Upconversion of YouTube Soccer videos to 3D - Camera Calibration - Player Analysis (detection/segmentation/tracking) - Player Depth estimation - Temporal Game Reconstruction
Scripts for getting training data from video games - Electronic Arts FIFA 2016 RenderDoc Depth buffer capture - Depth buffer to point cloud
Visualization tools - Example Unity project - Hololens VS Solution
These are the non "pip3 install" dependencies: - Detectron for detection and (instance) segmentation. - OpenPose for the players' pose estimation. - OpenCV 3.1 + OpenCV_contrib for image loading/edge estimation etc. I followed this guide and it worked fine. - Eigen3 for the instance segmentation - Cocoapi for mask utilities - Boost - CMake
The pipeline contains parts written or depending on python2, python3, cython, C/C++, which makes it a bit difficult to combine everything in one system. Therefore we break it into individual parts that have specific inputs and outputs (eg png to pickle) and communicate through a python3 class that reads, processes and writes the intermediate and final results.
First, download the repo and install its dependencies ```
git clone https://github.com/krematas/soccerontable $SOCCERCODE pip3 install -r requirements.txt
Let's start by downloading an example datasetwget http://grail.cs.washington.edu/projects/soccer/barcelona.zip unzip barcelona.zip
barcelona ├── images ├── 00000.jpg ├── 00001.jpg ├── ...
The original video was cropped from [YouTube](https://www.youtube.com/watch?v=hYU51XQruq0) and frames were extracted with avconv.Run Detectron to get bounding boxes and segmentation masks
mkdir $DATADIR/detectron
cp utils/thirdpartyscripts/infersubimages.py ./$DETECTRON/tools/ cd $DETECTRON python2 tools/infersubimages.py --cfg configs/122017baselines/e2emaskrcnnR-101-FPN2x.yaml --output-dir $DATADIR/detectron --image-ext jpg --wts https://s3-us-west-2.amazonaws.com/detectron/35861858/122017baselines/e2emaskrcnnR-101-FPN2x.yaml.023251.SgT4y1cO/output/train/coco2014train:coco2014valminusminival/generalizedrcnn/modelfinal.pkl $DATADIR/images/ ```
Now we can run the calibration step. In the first frame we give 4 manual correspondences and afterwards the camera parameters are optimized to fit a synthetic 3D field to the lines in the image.
cd $SOCCERCODE python3 demo/calibrate_video.py --path_to_data $DATADIR
Next, we estimate poses, near the bounding boxes that Mask-RCNN gave. ```
export LDLIBRARYPATH=$LDLIBRARYPATH:/usr/local/cuda/lib64/ python3 demo/estimateposes.py --pathtodata $DATADIR --openposedir $OPENPOSEDIR ```
The estimated poses cover very well the players in terms of localization/extend/etc. We use them to make individual crops of players for every frame for further processing. We use also the poses to refine the instance segmentation.
python3 demo/crop_players.py --path_to_data $DATADIR export OMP_NUM_THREADS=8 ./soccer3d/instancesegm/instancesegm --path_to_data $DATADIR/players/ --thresh 1.5 --path_to_model ./soccer3d/instancesegm/model.yml.gz
We combine the masks from Mask-RCNN and our pose-based optimization and we prepare the data for the network.
The model weights can be found here ```
python3 demo/combinemasksfornetwork.py --pathtodata $DATADIR python3 soccer3d/soccerdepth/test.py --pathto_data $DATADIR/players --modelpath $MODELPATH ```
Next, we convert tje estimated depthmaps to pointclouds.
python3 demo/depth_estimation_to_pointcloud.py --path_to_data $DATADIR
Finally we generate one mesh per frame, with the smooth position of the players, based on tracking. Note that the resolution of the mesh is reduced, so later it can easily fit into Hololens.
python3 demo/track_players.py --path_to_data $DATADIR python3 demo/generate_mesh.py --path_to_data $DATADIR
Just to be sure that everything is fine, we can have a simple opengl visualization ``` python3 demo/simplevisualization.py --pathto_data $DATADIR
The unity project and the hololens visual studio solution can be downloaded below (click on the image). The project was done on Unity version 2018.1 and to run the Hololens VS solution you will need a Hololens (with a bluetooth keyboard to place the hologram properly).