sac

by haarnoja

haarnoja / sac

Soft Actor-Critic

504 Stars 154 Forks Last release: Not found Other 362 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

This repository is no longer maintained. Please use our new Softlearning package instead.

Soft Actor-Critic

Soft actor-critic is a deep reinforcement learning framework for training maximum entropy policies in continuous domains. The algorithm is based on the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor presented at ICML 2018.

This implementation uses Tensorflow. For a PyTorch implementation of soft actor-critic, take a look at rlkit by Vitchyr Pong.

See the DIAYN documentation for using SAC for learning diverse skills.

Getting Started

Soft Actor-Critic can be run either locally or through Docker.

Prerequisites

You will need to have Docker and Docker Compose installed unless you want to run the environment locally.

Most of the models require a Mujoco license.

Docker installation

If you want to run the Mujoco environments, the docker environment needs to know where to find your Mujoco license key (

mjkey.txt
). You can either copy your key into
/.mujoco/mjkey.txt
, or you can specify the path to the key in your environment variables:
export MUJOCO_LICENSE_PATH=/mjkey.txt

Once that's done, you can run the Docker container with

docker-compose up

Docker compose creates a Docker container named

soft-actor-critic
and automatically sets the needed environment variables and volumes.

You can access the container with the typical Docker exec-command, i.e.

docker exec -it soft-actor-critic bash

See examples section for examples of how to train and simulate the agents.

To clean up the setup:

docker-compose down

Local installation

To get the environment installed correctly, you will first need to clone rllab, and have its path added to your PYTHONPATH environment variable.

  1. Clone rllab

    cd 
    git clone https://github.com/rll/rllab.git
    cd rllab
    git checkout b3a28992eca103cab3cb58363dd7a4bb07f250a0
    export PYTHONPATH=$(pwd):${PYTHONPATH}
    
  2. Download and copy mujoco files to rllab path: If you're running on OSX, download https://www.roboti.us/download/mjpro131osx.zip instead, and copy the

    .dylib
    files instead of
    .so
    files. ``` mkdir -p /tmp/mujoco
    tmp && cd /tmp/mujocotmp wget -P . https://www.roboti.us/download/mjpro131linux.zip unzip mjpro131linux.zip mkdir <installationpathofyourchoice>/rllab/vendor/mujoco cp ./mjpro131/bin/libmujoco131.so <installationpathofyourchoice>/rllab/vendor/mujoco cp ./mjpro131/bin/libglfw.so.3 <installationpathofyourchoice>/rllab/vendor/mujoco cd .. rm -rf /tmp/mujocotmp ```

  3. Copy your Mujoco license key (mjkey.txt) to rllab path:

    cp /mjkey.txt /rllab/vendor/mujoco
    
  4. Clone sac

    cd 
    git clone https://github.com/haarnoja/sac.git
    cd sac
    
  5. Create and activate conda environment

    cd sac
    conda env create -f environment.yml
    source activate sac
    

The environment should be ready to run. See examples section for examples of how to train and simulate the agents.

Finally, to deactivate and remove the conda environment:

source deactivate
conda remove --name sac --all

Examples

Training and simulating an agent

  1. To train the agent

    python ./examples/mujoco_all_sac.py --env=swimmer --log_dir="/root/sac/data/swimmer-experiment"
    
  2. To simulate the agent (NOTE: This step currently fails with the Docker installation, due to missing display.)

    python ./scripts/sim_policy.py /root/sac/data/swimmer-experiment/itr_.pkl
    

mujoco_all_sac.py
contains several different environments and there are more example scripts available in the
/examples
folder. For more information about the agents and configurations, run the scripts with
--help
flag. For example:
python ./examples/mujoco_all_sac.py --help
usage: mujoco_all_sac.py [-h]
                         [--env {ant,walker,swimmer,half-cheetah,humanoid,hopper}]
                         [--exp_name EXP_NAME] [--mode MODE]
                         [--log_dir LOG_DIR]

mujoco_all_sac.py
contains several different environments and there are more example scripts available in the
/examples
folder. For more information about the agents and configurations, run the scripts with
--help
flag. For example:
python ./examples/mujoco_all_sac.py --help
usage: mujoco_all_sac.py [-h]
                         [--env {ant,walker,swimmer,half-cheetah,humanoid,hopper}]
                         [--exp_name EXP_NAME] [--mode MODE]
                         [--log_dir LOG_DIR]

Benchmark Results

Benchmark results for some of the OpenAI Gym v2 environments can be found here.

Credits

The soft actor-critic algorithm was developed by Tuomas Haarnoja under the supervision of Prof. Sergey Levine and Prof. Pieter Abbeel at UC Berkeley. Special thanks to Vitchyr Pong, who wrote some parts of the code, and Kristian Hartikainen who helped testing, documenting, and polishing the code and streamlining the installation process. The work was supported by Berkeley Deep Drive.

Reference

@article{haarnoja2017soft,
  title={Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor},
  author={Haarnoja, Tuomas and Zhou, Aurick and Abbeel, Pieter and Levine, Sergey},
  booktitle={Deep Reinforcement Learning Symposium},
  year={2017}
}

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.