Need help with surreal?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

443 Stars 68 Forks MIT License 1.4K Commits 7 Opened issues


Open-Source Distributed Reinforcement Learning Framework by Stanford Vision and Learning Lab

Services available


Need anything else?

Contributors list



Open-Source Distributed Reinforcement Learning Framework

Stanford Vision and Learning Lab

SURREAL is a fully integrated framework that runs state-of-the-art distributed reinforcement learning (RL) algorithms.

  • Scalability. RL algorithms are data hungry by nature. Even the simplest Atari games, like Breakout, typically requires up to a billion frames to learn a good solution. To accelerate training significantly, SURREAL parallelizes the environment simulation and learning. The system can easily scale to thousands of CPUs and hundreds of GPUs.

  • Flexibility. SURREAL unifies distributed on-policy and off-policy learning into a single algorithmic formulation. The key is to separate experience generation from learning. Parallel actors generate massive amount of experience data, while a single, centralized learner performs model updates. Each actor interacts with the environment independently, which allows them to diversify the exploration for hard long-horizon robotic tasks. They send the experiences to a centralized buffer, which can be instantiated as a FIFO queue for on-policy mode and replay memory for off-policy mode.

  • Reproducibility. RL algorithms are notoriously hard to reproduce [Henderson et al., 2017], due to multiple sources of variations like algorithm implementation details, library dependencies, and hardware types. We address this by providing an end-to-end integrated pipeline that replicates our full cluster hardware and software runtime setup.


Surreal algorithms can be deployed at various scales. It can run on a single laptop and solve easier locomotion tasks, or run on hundreds of machines to solve complex manipulation tasks.
* Surreal on your Laptop * Surreal on Google Cloud Kubenetes Engine
* Customizing Surreal
* Documentation Index


  • Scalability of Surreal-PPO with up to 1024 actors on Surreal Robotics Suite.

  • Training curves of 16 actors on OpenAI Gym tasks for 3 hours, compared to other baselines.


Please cite our CORL paper if you use this repository in your publications:

  title={SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark},
  author={Fan, Linxi and Zhu, Yuke and Zhu, Jiren and Liu, Zihua and Zeng, Orien and Gupta, Anchit and Creus-Costa, Joan and Savarese, Silvio and Fei-Fei, Li},
  booktitle={Conference on Robot Learning},

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.