Need help with surreal?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.


Open-Source Distributed Reinforcement Learning Framework by Stanford Vision and Learning Lab

428 Stars 67 Forks MIT License 1.4K Commits 7 Opened issues

Services available

Need anything else?



Open-Source Distributed Reinforcement Learning Framework

Stanford Vision and Learning Lab

SURREAL is a fully integrated framework that runs state-of-the-art distributed reinforcement learning (RL) algorithms.

  • Scalability. RL algorithms are data hungry by nature. Even the simplest Atari games, like Breakout, typically requires up to a billion frames to learn a good solution. To accelerate training significantly, SURREAL parallelizes the environment simulation and learning. The system can easily scale to thousands of CPUs and hundreds of GPUs.

  • Flexibility. SURREAL unifies distributed on-policy and off-policy learning into a single algorithmic formulation. The key is to separate experience generation from learning. Parallel actors generate massive amount of experience data, while a single, centralized learner performs model updates. Each actor interacts with the environment independently, which allows them to diversify the exploration for hard long-horizon robotic tasks. They send the experiences to a centralized buffer, which can be instantiated as a FIFO queue for on-policy mode and replay memory for off-policy mode.

  • Reproducibility. RL algorithms are notoriously hard to reproduce [Henderson et al., 2017], due to multiple sources of variations like algorithm implementation details, library dependencies, and hardware types. We address this by providing an end-to-end integrated pipeline that replicates our full cluster hardware and software runtime setup.


Surreal algorithms can be deployed at various scales. It can run on a single laptop and solve easier locomotion tasks, or run on hundreds of machines to solve complex manipulation tasks.
* Surreal on your Laptop * Surreal on Google Cloud Kubenetes Engine
* Customizing Surreal
* Documentation Index


  • Scalability of Surreal-PPO with up to 1024 actors on Surreal Robotics Suite.

  • Training curves of 16 actors on OpenAI Gym tasks for 3 hours, compared to other baselines.


Please cite our CORL paper if you use this repository in your publications:

  title={SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark},
  author={Fan, Linxi and Zhu, Yuke and Zhu, Jiren and Liu, Zihua and Zeng, Orien and Gupta, Anchit and Creus-Costa, Joan and Savarese, Silvio and Fei-Fei, Li},
  booktitle={Conference on Robot Learning},

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.