Need help with DialogRPT?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

250 Stars 19 Forks MIT License 106 Commits 5 Opened issues


EMNLP 2020: "Dialogue Response Ranking Training with Large-Scale Human Feedback Data"

Services available


Need anything else?

Contributors list

# 77,167
104 commits

DialogRPT: Dialog Ranking Pretrained Transformers

DialogRPT predicts human feedback (upvotes👍 or replies💬) of dialogue responses.

It is a set of dialog response ranking models proposed by Microsoft Research NLP Group trained on 100 + millions of human feedback data, accepted to appear at EMNLP'20. It can be used to improve existing dialog generation model (e.g., DialoGPT) by re-ranking the generated response candidates. This repo provides a PyTorch implementation and pretrained models.

Quick links: * Paper * Intro Talk and Slides * Demo: original or HuggingFace * Dataset

We considered the following tasks and provided corresponding pretrained models. (Click 💾 to download original pytorch checkpoint for this repo, or click 🤗 to use HuggingFace model card)

| Task | Description | Pretrained model | | :------------- | :----------- | :-----------: | | Human feedback | | | |

| How likely the response gets the most upvotes? | 💾 / 🤗 | |
| How likely the response gets the most direct replies? | 💾 / 🤗 | |
| How likely the response gets the longest follow-up thread? | 💾 / 🤗 | | Human-like (human vs fake) | | | |
| How relevant the response is for the given context? | 💾 / 🤗 | |
| How likely the response is human-written rather than machine-generated? | 💾 / 🤗 |


Quick Start


Option 1: run locally

git clone
cd DialogRPT
conda create -n dialogrpt python=3.6
conda activate dialogrpt
pip install -r requirements.txt

Option 2: run on Colab Notebook. You can either use Demo (original) or Demo (HuggingFace)

Use rankers only

In the following example, the model predicts that, given the same context "I love NLP!", response "Here’s a free textbook (URL) in case anyone needs it." is gets more upvotes than response "Me too!". ```bash python src/ play -p=restore/updown.pth

Context: I love NLP!

Response: Here’s a free textbook (URL) in case anyone needs it.

score = 0.613

Context: I love NLP!

Response: Me too!

score = 0.111

You can also play the ensemble model, which involves multiple models defined in its [config file](restore/ensemble.yml) (see this file for details). 
python src/ play -p=restore/ensemble.yml

To score a list of (context, response) pairs, please provide a input file (

), which is tab-separated in format
context \t response0 \t response1 ...
. See example input file * Using a single ranker (see expected output) ```bash python src/ test --data=doc/toy.tsv -p=restore/updown.pth

downloading pretrained model to restore/updown.pth

100% [....................] 1520029114 / 1520029114

loading from restore/updown.pth

ranking doc/toy.tsv

totally processed 2 line, avghypscore 0.264, tophypscore 0.409

results saved to doc/toy.tsv.ranked.jsonl

* Using an ensemble model (see [expected output](
python src/ test --data=doc/toy.tsv -p=restore/ensemble.yml

Statistics of the scoring results can be shown with the following command, e.g. for

```bash python src/ stats --data=doc/toy.tsv.ensemble.jsonl

|best |avg


_score |0.339 |0.206

humanvsrand |0.928 |0.861

humanvsmachine |0.575 |0.525

updown |0.409 |0.264

depth |0.304 |0.153

width |0.225 |0.114

final |0.339 |0.206


n_cxt: 2

avg n_hyp per cxt: 2.50

Use generator + ranker

Dialog generation models can be improved by integrating with the response ranking models. For example, given the context "Can we restart 2020?", DialoGPT may return the following responses by sampling decoding (or you can try beam search without --sampling). Some of them, e.g., "Yes, we can." has a high generation probability (gen 0.496), but less interesting (ranker 0.302). So the rankers will put in position lower than ones more likely to be upvoted, e.g. "I think we should go back to the beginning, and start from the beginning." which is relatively less likely to be generated (gen 0.383) but seems more interesting (ranker 0.431)

```bash python src/ play -pg=restore/medium_ft.pkl -pr=restore/updown.pth --sampling #

Context: Can we restart 2020?

0.431 gen 0.383 ranker 0.431 I think we should go back to the beginning, and start from the beginning.

0.429 gen 0.227 ranker 0.429 I think I'll just sit here and wait for 2020

0.377 gen 0.249 ranker 0.377 Yeah, let's just start from the beginning

0.323 gen 0.195 ranker 0.323 I think we should just give up and let the year just pass.

0.304 gen 0.395 ranker 0.304 Yes. We can.

0.302 gen 0.496 ranker 0.302 Yes, we can.

0.283 gen 0.351 ranker 0.283 It's been a while since we've seen a good reboot.

0.174 gen 0.306 ranker 0.174 I'm up for it

0.168 gen 0.463 ranker 0.168 I'm down

0.153 gen 0.328 ranker 0.153 I think so, yes.


Similarly, you can use the ensemble model.

python src/ -pg=restore/medium_ft.pkl -pr=restore/ensemble.yml
To generate from a list of contexts stored in a line-separated file, provide it with
and use the command below:
python src/ test --path_test=path/to/list/of/contexts -pg=restore/medium_ft.pkl -pr=restore/ensemble.yml


Traning dataset can be built with this script, which downloads raw data from a third party dump and extract comparable pairs of comments for classification tasks.

Testing data can be downloaded here use the command below
Please checkout our Dataset webpage for data examples, description, statistics and more.


We use DialoGPT to initialize the model. Please download with

wget -P restore
For the human feedback prediction tasks, we specify
to only validate on less-noisy samples (not applied to training).
python src/ train --data=data/out/updown -p=restore/medium_ft.pkl --min_score_gap=20 --min_rank_gap=0.5
python src/ train --data=data/out/depth -p=restore/medium_ft.pkl --min_score_gap=4 --min_rank_gap=0.5
python src/ train --data=data/out/width -p=restore/medium_ft.pkl --min_score_gap=4 --min_rank_gap=0.5
task, use the
flag to feed rand human response as negative examples. We can reuse previous dataset (e.g.
python src/ train --data=data/out/updown -p=restore/medium_ft.pkl --mismatch
task, we build dataset by pair human response with a response generated by DialoGPT with topk decoding
python src/ train --data=data/out/human_vs_machine -p=restore/medium_ft.pkl

We trained all models on a Nvidia V100 4-core GPU (each core with 32G memory) with the following hyperparameters. Checkpoint with the best validation accuracy is used as final model. | Argument | Value | Description | | :------------- | :-----------: |:------------- | |

| 256 | total batch size for all GPUs. | |
| 1024 | number of samples used for validation (i.e. dev set size). | |
| 3e-05 | learning rate | |
| 50 | max allowed sequence length.
if longer, leading tokens will be truncated | |
| 1 | max allowed hour difference between positive and negative samples.
If longer, this pair will be discarded for train/vali|


Human feedback prediction

The performance on

, and
can be measured with the following commands, respectively. The
arguments are consistent with the values used to measure validation loss during training.
python src/ eval_human_feedback -p=restore/updown.pth --data=test/human_feedback/updown.tsv --min_score_gap=20 --min_rank_gap=0.5
python src/ eval_human_feedback -p=restore/depth.pth --data=test/human_feedback/depth.tsv --min_score_gap=4 --min_rank_gap=0.5
python src/ eval_human_feedback -p=restore/width.pth --data=test/human_feedback/width.tsv --min_score_gap=4 --min_rank_gap=0.5

The expected pairwise accuracy on 5000 test samples is listed in the table below (from Table 5 of the paper). Note even by random guess one can get accuracy of 0.500. | human feedback |

| | :------------- | :------: |:------------: |:--------: | | Dialog ppl. | 0.488 | 0.508 | 0.513 | | Reverse dialog ppl. | 0.560 | 0.557 | 0.571 | | DialogRPT (ours)| 0.683 | 0.695 | 0.752 |

Human-like classification

  • human_vs_rand
    task: Although the model is trained on
    corpus only, we measured its zero-shot performance on several unseen corpora (
    python src/ eval_human_vs_rand -p=restore/human_vs_rand.pth --data=test/human_vs_fake/reddit
    python src/ eval_human_vs_rand -p=restore/human_vs_rand.pth --data=test/human_vs_fake/dailydialog
    python src/ eval_human_vs_rand -p=restore/human_vs_rand.pth --data=test/human_vs_fake/twitter
    python src/ eval_human_vs_rand -p=restore/human_vs_rand.pth --data=test/human_vs_fake/personachat
    The expected
    [email protected]
    metric on 5000 test samples is listed in the table below (from Table 7 of the paper).
    [email protected]
    measures, for the same context, given
    positive responses and
    negative responses, how many positive responses are in top-
    of the ranked responses. |
    | | :------------- | :------: |:------------: |:--------: |:------------: | | BM25 | 0.309 | 0.182 | 0.178 | 0.117 | | Dialog ppl. | 0.560 | 0.176 | 0.107 | 0.108 | | Reverse dialog ppl. | 0.775 | 0.457 | 0.440 | 0.449 | | ConveRT | 0.760 | 0.380 | 0.439 | 0.197 | | DialogRPT (ours)| 0.886 | 0.621 | 0.548 | 0.479 |
  • human_vs_machine
    task: its performance is only evaluated for
    corpus. ```bash python src/ --task=evalhumanvsmachine -p=restore/humanvsmachine.pth --data=test/humanvs_fake/reddit

    expecting accuracy ~0.98



If you use our dataset or model, please cite our paper

    title={Dialogue Response RankingTraining with Large-Scale Human Feedback Data},
    author={Xiang Gao and Yizhe Zhang and Michel Galley and Chris Brockett and Bill Dolan},

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.