Need help with Microsoft-Rocket-Video-Analytics-Platform?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

microsoft
160 Stars 51 Forks MIT License 31 Commits 5 Opened issues

Description

A highly extensible software stack to empower everyone to build practical real-world live video analytics applications for object detection and counting with cutting edge machine learning algorithms.

Services available

!
?

Need anything else?

Contributors list

# 2,497
roslyn
.NET
PHP
parser-...
14 commits

Microsoft Rocket Video Analytics Platform

A highly extensible software stack to empower everyone to build practical real-world live video analytics applications for object detection and counting/alerting with cutting edge machine learning algorithms. The repository features a hybrid edge-cloud video analytics pipeline (built on C# .NET Core), which allows TensorFlow DNN model plug-in, GPU/FPGA acceleration, docker containerization/Kubernetes orchestration, and interactive querying for after-the-fact analysis. A brief summary of Rocket platform can be found inside :memo:Rocket-features-and-pipelines.pdf.

Feel free to check out our :memo:webinar on Rocket from Dec 2019.

How to run the code

Step 1: Set up environment

Setup on Windows

  • Microsoft Visual Studio (VS 2017 is preferred) is recommended IDE for Rocket on Windows 10. While installing Visual Studio, please also add C++ 2015.3 v14.00 (v140) toolset to your local machine. Snapshot below shows how to include C++ 2015.3 v14.00 from Visual Studio Installer.
    C++v140

  • Follow instructions to install .NET Core 2.2 (2.2.102 is preferred).

  • To enable GPU support, install CUDA Toolkit and cuDNN. Please also make sure your NVIDIA driver is up-to-date.

    • CUDA 8.0 (e.g., cuda8.0.61win10_network.exe) is needed for Darknet (e.g., YOLO) models.

      After installation, please make sure files in

      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\extras\visual_studio_integration\MSBuildExtensions
      are copied to
      C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\BuildCustomizations
    • CUDA 9.1 (e.g., cuda9.1.85win10_network.exe) is needed to support TensorFlow models.

    • cuDNN v7 is preferred (e.g., cudnn-8.0-windows10-x64-v7.2.1.38.zip).

      Copy

      \cuda\bin\cudnn64_7.dll
      to
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin
      .
      Copy
      \cuda\ include\cudnn.h
      to
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include
      .
      Copy
      \cuda\lib\x64\cudnn.lib
      to
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64
      .
      Add Variable Name:
      CUDA_PATH
      with Variable Value:
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
      into Environment Variables.
      PathVariable
    • Restart your computer after installing CUDA and cuDNN.

Setup on Linux

Docker is recommended to run Rocket on Linux. Below we use Ubuntu 16.04 as an example to walk through the steps of building Rocket docker image and run it with GPU acceleration.

  • Install .NET Core 2.2 SDK (2.2.301 is preferred).
  • Install docker-ce (version 18.09.7 is preferred).
  • Install NVIDIA driver based on your GPU model (e.g., 418.67 for Tesla GPU).
  • Install nvidia-docker2. The NVIDIA Container Toolkit allows users to build and run GPU accelerated Docker containers. Note that you do NOT need to install the CUDA toolkit on the host, but GPU driver needs to be installed.

Step 2: Run the pipeline

Check out the repository.

Prepare video feeds and line configuration

  • Prepare video feeds. Rocket can be fed with either live video streams (e.g.,
    rtsp://:/
    ) or local video files (should be put into
    \media\
    ). A sample video file
    sample.mp4
    is already included in
    \media\
    .
  • Prepare a configuration file (should be placed into
    \cfg\
    ) used in line-based counting/alerting and cascaded DNN calls. Each line in the file defines a line-of-interest with the format below.
          

    A line configuration file
    sample.txt
    manually created based on
    sample.mp4
    is also included in the folder
    \cfg\
    . sampleline

Build on Windows

  • Run
    Config.bat
    before the first time you run Rocket to download pre-compiled OpenCV and TensorFlow binaries as well as Darknet YOLO weights files. It may take few minutes depending on your network status. Proceed only when all downloads finish. YOLOv3 and Tiny YOLOv3 are already included in Rocket. You can plug-in other YOLO models as you wish.
  • Launch
    VAP.sln
    in
    src\VAP\
    from Visual Studio.
  • Set pipeline config

    PplConfig
    in VideoPipelineCore - App.config. We have pre-compiled six configurations in the code. Pipeline descriptions are also included in :memo:Rocket-features-and-pipelines.pdf.
    • 0: Line-based counting
    • 1: Darknet Yolo v3 on every frame (slide #7)
    • 2: TensorFlow FastRCNN on every frame (slide #8)
    • 3: Background subtraction-based (BGS) early filtering -> Darknet Tiny Yolo -> Darknet Yolo v3 (slide #9)
    • 4: BGS early filtering -> Darknet Tiny Yolo -> Database (ArangoDB and blob storage on Azure) (slide #10)
    • 5: BGS early filtering -> TensorFlow Fast R-CNN -> Azure Machine Learning (cloud) (slide #11)
  • (Optional) Set up your own database and Azure Machine Learning service if

    PplConfig
    is set to 4 or 5.
    • Azure Database:
      • Deploy SQL database like MySQL or NoSQL database such as ArangoDB on Azure by creating a VM.
      • Supply database settings (e.g., server name, user name, credentials etc.) to Rocket in
        App.Config
        .
      • You can also set up your cloud storage (e.g., Azure Blob Storage) to store images/videos. In pipeline 4, Rocket sends detection images to an Azure storage account and metadata to an Azure database. azureconfig
    • Azure Machine Learning:
      • Deploy your deep learning models to Azure (e.g., using Azure Kubernetes Service or AKS) for inference with GPU or FPGA.
      • After deploying your model as a web service, provide host URL, key, and service ID to VAP in
        App.Config
        . Rocker will handle the communication between local modules and the cloud service. azureconfig
  • Build the solution.

  • Run the code.

    • Using Visual Studio: set VideoPipelineCore - Property - Debug - Application Arguments
          
      . To run Rocket on the sample video, for example, arguments can be set to
      sample.mp4 sample.txt 1 1 car
      .
    • Using Command Line (CMD or PowerShell): run
      dotnet .\VideoPipelineCore.dll     
      in
      \src\VAP\VideoPipelineCore\bin\Debug\netcoreapp2.2
      . For instance,
      dotnet .\VideoPipelineCore.dll sample.mp4 sample.txt 1 1 car
      .

Build on Linux

We have pre-built a Rocket docker image from docker branch with local processing only (slide #12 without cloud parts). The image is hosted on Docker Hub, a public library and community for container images, and you will be asked to login before pull/push images (sign up first if you don't have an account).

To test on the pre-built Rocket image, run

docker pull ycshu086/rocket-sample-edgeonly:0.1

Once pulled, run the command below to start Rocket with NVIDIA GPU.

docker run --runtime=nvidia -v :/app/output ycshu086/rocket-sample-edgeonly:0.1 sample.mp4 sample.txt 1 1 car
  • Build your own Rocket pipeline on Linux

    • Pull base docker image with CUDA toolkit and OpenCV. This image is needed to build Rocket docker image.
      docker pull ycshu086/ubuntu-dotnetcore-opencv-opencvsharp-cuda-cudnn:
      .
    • Git clone docker branch for source code to dockerize Rocket on Linux.
    • Create line configuration file(s) inside
      \cfg
      . If you are running Rocket on a pre-recorded video, please also copy the video file into
      \media
      .
    • (Optional) Update
      \src\VAP\VideoPipelineCore\App.Config
      to set proper parameters for database and Azure Machine Learning service connection.
    • Run
      sudo chmod 744 Config.sh
      and
      sudo ./Config.sh
      before the first time you build Rocket image to download pre-compiled TensorFlow binaries.
    • Run
      docker build
      to build Rocket image using
      Dockerfile.VAP
      .
      docker build -t /: -f Dockerfile.VAP .
    • (Optional) Push Rocket image to a cloud repository (e.g., docker hub, Azure Container Registry etc.) if you need to run it somewhere else.
      docker push -t /:
  • Run Rocket image on Linux

    • Pull a pre-built Rocket docker image to the local machine. You can use
      docker images
      to check existing images.
      docker pull -t /:

    • Mount volume into the container and run Rocket image with NVIDIA GPU.
      docker run --runtime=nvidia -v :/app/output /: sample.mp4 sample.txt 1 1 car

Step 3: Results

Output images are generated in folders in

\src\VAP\VideoPipelineCore\bin\
(Windows), or the local directory you mount during
docker run
on Linux. Results from different modules are sent to different directories (e.g.,
output_bgsline
for background subtraction-based detector) whereas
output_all
has images from all modules. Name of each file consists of frame ID, module name, and confidence score. Below are few sample results from running pipeline 3 and pipeline 5 on
sample.mp4
. You should also see results printed in console during running. output The above illustration on pipeline 3 shows that at frame 2679, background subtraction detected an object, tiny Yolo DNN confirmed it was a car with a confidence of 0.24, and heavy Yolo v3 confirmed it with a confidence of 0.92. Likewise, for pipeline 5 where the TensorFlow FastRNN model had a confidence of 0.55 and AzureML (in the cloud) came back with a confidence of 0.76 for the same object.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.