by facebookresearch

facebookresearch / minirts

We release dataset collected for our research, code that implement neural network models described i...

130 Stars 28 Forks Last release: Not found Other 13 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Hierarchical Decision Making by Generating and Following Natural Language Instructions

This is the repo for paper Hierarchical Decision Making by Generating and Following Natural Language Instructions.


conda install lua numpy tqdm
conda install -c conda-forge tensorboardx

Get Started

Clone repo

git clone ...
git submodule sync && git submodule update --init --recursive

Download dataset & pretrained models

To downalod and unzip the original replays, processed json files, and dataset, from the following command. Note that it will take a while for the command to finish.

cd data

To download some pretrained models used in the paper:

cd pretrained_models

Visualize dataset

We build a visualization tool that works directly with json file so that people can get a more intuitive view of the dataset and start working on it without compiling the game. Please go to the visual folder for detailed instructions on how to use it.

Train models

We put the shell scripts that can be used to re-train the model with configurations used in the paper in

. Simply run command like
sh scripts/

to start training. The command needs to be run under

folder. Normally it will take quite a while to load the dataset. For quick testing and debugging, one can add
at the end of the shell script to use the dev dataset instead, which contains only 2000 entries and thus much faster to load.

Run matches between models

To run matches between trained models, we first need to compile the game. Please see the "Build" and "Set env var" section for details. After the game is compiled, the following command can be used to launch matches between an

RNN coach + RNN
zero executor
(the one that does not use latent language).
python --coach1 rnn500 --executor1 rnn \
        --coach2 rnn500 --executor2 zero \
        --num_thread 500 --seed 9999



This is the main folder for our algorithm, containing code for data processing, model definition & training, and evaluation. See the readme file for each subfolder for more details.


This contains a web tool for visualizing dataset from json so that we can have a peek of the dataset without compiling the game.


This folder contains the implementation of the game, including game logic, some built-in AIs used for collecting data, as well as necessary backends to extract features from game state for model evaluation.


This folder defines a set of infra that dynamically batches data from various C++ game threads and transfer them between C++ and Python.


mkdir build
cd build
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
cmake ..

Set Env Variables

Note that we need to set the following before running any multi-threading program that uses the C++ torch::Tensor. Otherwise a simple tensor operation will use all cores by default.


Control Executor with Text

We can control the executor ourselves by inputting text command to the trained executor. First we need to set up the web server for the backend so that we can watch the gameplay in browser while controlling the executor.

We provide a script to install apache without root access. If you have root privilege, you can simply run

sudo apt-get update & sudo apt-get
install apache2
After installation finishes, edit
to change the
Listen 80
(line52) to
Listen 8000
or any number >1024. The reason is that the ports with lower numbers are reserved by system and requires sudo to use them.

Then we need to link our frontend code to the apache root directory & start server

ln -s $PWD/game/frontend $PWD/apache/httpd/htdocs/game
cd apache/httpd
./bin/apachectl start

Now open a browser and navigate to

. You should see
It Works
. Otherwise there are some issue with the server set up.

Then we can start a human game! ```bash cd ROOT/scripts/behaviorclone python --resource 500 --verbose

it should show 'Waiting for websocket client ...'

On the browser, navigate to
and wait for the model to be loaded. The command line will prompt the
top 500 instructions the model was trained on. If you are using RNN
executor (by default), you don't have to choose from these
instructions as the RNN can ideally handle unseen combinations. If you
are using OneHot executor, you should input an instruction from the


If you use this repo in your research, please consider citing the paper as follows:

@article{DBLP:journals/corr/abs-1906-00744, author = {Hengyuan Hu and Denis Yarats and Qucheng Gong and Yuandong Tian and Mike Lewis}, title = {Hierarchical Decision Making by Generating and Following Natural Language Instructions}, journal = {CoRR}, volume = {abs/1906.00744}, year = {2019}, url = {}, archivePrefix = {arXiv}, eprint = {1906.00744}, timestamp = {Thu, 13 Jun 2019 13:36:00 +0200}, biburl = {}, bibsource = {dblp computer science bibliography,} } ```


Copyright (c) Facebook, Inc. and its affiliates. All rights reserved.

This source code is licensed under the license found in the LICENSE file in the root directory of this source tree.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.