Need help with DCL?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

chingyaoc
141 Stars 15 Forks 12 Commits 0 Opened issues

Description

NeurIPS 2020, Debiased Contrastive Learning

Services available

!
?

Need anything else?

Contributors list

# 111,567
vqa
Python
pytorch
Shell
9 commits
# 459,894
HTML
Keras
pytorch
caffe2
1 commit

Debiased Contrastive Learning

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels.

Debiased Contrastive Learning NeurIPS 2020 [paper]
Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka

Prerequisites

  • Python 3.7
  • PyTorch 1.3.1
  • PIL
  • OpenCV

Contrastive Representation Learning

We can train standard (biased) or debiased version (M=1) of SimCLR with

main.py
on STL10 dataset.

flags: -

--debiased
: use debiased objective (True) or standard objective (False) -
--tau_plus
: specify class probability -
--batch_size
: batch size for SimCLR

For instance, run the following command to train a debiased encoder.

python main.py --tau_plus = 0.1

Linear evaluation

The model is evaluated by training a linear classifier after fixing the learned embedding.

path flags: -

--model_path
: specify the path to saved model
python linear.py --model_path results/model_400.pth

Pretrained Models

| | tauplus | Arch | Latent Dim | Batch Size | Accuracy(%) | Download | |----------|:---:|:----:|:---:|:---:|:---:|:---:| | Biased | tauplus = 0.0 | ResNet50 | 128 | 256 | 80.15 | model| | Debiased |tauplus = 0.05 | ResNet50 | 128 | 256 | 81.85 | model| | Debiased |tauplus = 0.1 | ResNet50 | 128 | 256 | 84.26 | model|

Citation

If you find this repo useful for your research, please consider citing the paper

@article{chuang2020debiased,
  title={Debiased contrastive learning},
  author={Chuang, Ching-Yao and Robinson, Joshua and Lin, Yen-Chen and Torralba, Antonio and Jegelka, Stefanie},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

For any questions, please contact Ching-Yao Chuang ([email protected]).

Acknowledgements

Part of this code is inspired by leftthomas/SimCLR.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.