Need help with Google_Refexp_toolbox?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

144 Stars 50 Forks 11 Commits 3 Opened issues


The toolbox for the Google Refexp dataset proposed in this paper:

Services available


Need anything else?

Contributors list

# 311,942
10 commits
# 5,562
1 commit

Python toolbox for the Google Referring Expressions Dataset

The Google RefExp dataset is a collection of text descriptions of objects in images which builds on the publicly available MS-COCO dataset. Whereas the image captions in MS-COCO apply to the entire image, this dataset focuses on text descriptions that allow one to uniquely identify a single object or region within an image. See more details in this paper: Generation and Comprehension of Unambiguous Object Descriptions

Sample of the data

Green dot is the object that is being referred to. Sentences are generated by humans in a way that uniquely describes the chosen object.

Mountain View Mountain View Mountain View
  • A girl wearing glasses and a pink shirt.
  • An Asian girl with a pink shirt eating at the table.
  • A boy brushing his hair while looking at his reflection.
  • A young male child in pajamas shaking around a hairbrush in the mirror.
  • Zebra looking towards the camera.
  • A zebra third from the left.


  • python 2.7 (Need numpy, scipy, matlabplot, PIL packages. All included in Anaconda)

Setup and data downloading

Easy setup


Running the script will do the following five things:

  1. Download Google Refexp Data
  2. Download and compile the COCO toolbox
  3. Download COCO annotations
  4. Download COCO images
  5. Align the Google Refexp Data with COCO annotations

At each step you will be prompted by keyboard whether or not you would like to skip a step. Note that the MS COCO images (13GB) and annotations (158MB) are very large and it takes some time to download all of them.

Manual downloading and setup

You can download the GoogleRefexp data directly from this link.

If you have already played with MS COCO and do not want to have two copies of them, you can choose to create a symbolic link from external to your COCO toolkit. E.g.:

  ln -sf $YOUR_PATH_TO_COCO_TOOLBOX ./external/coco

Please make sure the following on are on your path:

  1. compiled PythonAPI at external/coco/PythonAPI
  2. the annotation file at external/coco/annotations/instances_train2014.json
  3. COCO images at external/coco/images/train2014/.

You can create symbolic links if you have already downloaded the data and compiled the COCO toolbox.

Then run to download the Google Refexp data and compile this toolbox. You can skip steps 2, 3, 4.


For visualization and utility functions, please see googlerefexpdataset_demo.ipynb.

For automatic and Amazon Mechanical Turk (AMT) evaluation of the comprehension and generation tasks, please see googlerefexpeval_demo.ipynb; The appropriate output format for a comprehension/generation algorithm is described in ./evaluation/ and ./evaluation/

We also provide two sample outputs for reference. For the comprehension task, we use a naive baseline which is a random shuffle of the region candidates (./evaluation/sampleresults/sampleresultscomprehension.json). For the generation task, we use a naive baseline which outputs the class name of an object (./evaluation/sampleresults/sampleresultsgeneration.json).

If you are not familiar with AMT evaluations, please see this tutorial The interface and APIs provided by this toolbox have already grouped 5 evaluations into one HIT. In our experiment, paying 2 cents for one HIT leads to reasonable results.


If you find the dataset and toolbox useful in your research, please consider citing:

  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin},


This data is released by Google under the following license: Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Toolbox Developers

Junhua Mao and Oana Camburu

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.