Overview

WILDS is a benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications, from tumor identification to wildlife monitoring to poverty mapping.

The WILDS package contains: 1. Data loaders that automatically handle data downloading, processing, and splitting, and 2. Dataset evaluators that standardize model evaluation for each dataset.

In addition, the example scripts contain default models, allowing new algorithms to be easily added and run on all of the WILDS datasets.

For more information, please read our paper or visit our website. For questions and feedback, please post on the discussion board.

Installation

We recommend using pip to install WILDS:

bash
pip install wilds

If you have already installed it, please check that you have the latest version: ```bash python -c "import wilds; print(wilds.version)"

This should print "1.0.0". If it doesn't, update by running:

pip install -U wilds ```

If you plan to edit or contribute to WILDS, you should install from source:

bash
git clone [email protected]:p-lambda/wilds.git
cd wilds
pip install -e .

Requirements

• numpy>=1.19.1
• pandas>=1.1.0
• pillow>=7.2.0
• torch>=1.7.0
• tqdm>=4.53.0
• pytz>=2020.4
• outdated>=0.2.0
• ogb>=1.2.3
• torch-scatter>=2.0.5
• torch-geometric>=1.6.1

Running

pip install wilds

pip install -e .

torch-scatter

torch-geometric

Default models

After installing the WILDS package, you can use the scripts in

examples/

bash
git clone [email protected]:p-lambda/wilds.git

To run these scripts, you will need to install these additional dependencies:

• torchvision>=0.8.1
• transformers>=3.5.0

All baseline experiments in the paper were run on Python 3.8.5 and CUDA 10.1.

Usage

Default models

In the

examples/

cd examples
python run_expt.py --dataset iwildcam --algorithm ERM --root_dir data
python run_expt.py --dataset civilcomments --algorithm groupDRO --root_dir data

The scripts are set up to facilitate general-purpose algorithm development: new algorithms can be added to

examples/algorithms

The first time you run these scripts, you might need to download the datasets. You can do so with the

--download

python run_expt.py --dataset civilcomments --algorithm groupDRO --root_dir data --download

Data loading

The WILDS package provides a simple, standardized interface for all datasets in the benchmark. This short Python snippet covers all of the steps of getting started with a WILDS dataset, including dataset download and initialization, accessing various splits, and preparing a user-customizable data loader.

>>> from wilds.datasets.iwildcam_dataset import IWildCamDataset
>>> from wilds.common.data_loaders import get_train_loader
>>> import torchvision.transforms as transforms

Load the full dataset, and download it if necessary
>>> dataset = IWildCamDataset(download=True)
Get the training set
>>> train_data = dataset.get_subset('train',
...                                 transform=transforms.Compose([transforms.Resize((224,224)),
...                                                               transforms.ToTensor()]))
Prepare the standard data loader
>>> train_loader = get_train_loader('standard', train_data, batch_size=16)
Train loop
>>> for x, y_true, metadata in train_loader:
...   ...

The

metadata

Domain information

To allow algorithms to leverage domain annotations as well as other groupings over the available metadata, the WILDS package provides

Grouper

Grouper

>>> from wilds.common.grouper import CombinatorialGrouper

Initialize grouper, which extracts domain information
In this example, we form domains based on location
>>> grouper = CombinatorialGrouper(dataset, ['location'])
Train loop
>>> for x, y_true, metadata in train_loader:
...   z = grouper.metadata_to_group(metadata)
...   ...

The

Grouper

# Prepare a group data loader that samples from user-specified groups
>>> train_loader = get_train_loader('group', train_data,
...                                 grouper=grouper,
...                                 n_groups_per_batch=2,
...                                 batch_size=16)

Evaluators

The WILDS package standardizes and automates evaluation for each dataset. Invoking the

eval

>>> from wilds.common.data_loaders import get_eval_loader

Get the test set
>>> test_data = dataset.get_subset('test',
...                                 transform=transforms.Compose([transforms.Resize((224,224)),
...                                                               transforms.ToTensor()]))
Prepare the data loader
>>> test_loader = get_eval_loader('standard', test_data, batch_size=16)
Get predictions for the full test set
>>> for x, y_true, metadata in test_loader:
...   y_pred = model(x)
...   [accumulate y_true, y_pred, metadata]
Evaluate
>>> dataset.eval(all_y_pred, all_y_true, all_metadata)
{'recall_macro_all': 0.66, ...}

Citing WILDS

If you use WILDS datasets in your work, please cite our paper (Bibtex):

• WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020). Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.

Please also cite the original papers that introduce the datasets, as listed on the datasets page.

Acknowledgements

The design of the WILDS benchmark was inspired by the Open Graph Benchmark, and we are grateful to the Open Graph Benchmark team for their advice and help in setting up WILDS.