Need help with BERT-Keyword-Extractor?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

162 Stars 45 Forks 15 Commits 11 Opened issues


Deep Keyphrase Extraction using BERT

Services available


Need anything else?

Contributors list

No Data

Deep Keyphrase Extraction using BERT

I have used BERT Token Classification Model to extract keywords from a sentence. Feel free to clone and use it. If you face any problems, kindly post it on issues section.

Special credits to BERT authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, original repo and Huggingface for PyTorch version original repo.


You need:

pytorch 1.0
python 3.6
pytorch-pretrained-bert 0.4.0


script can be used to extract keywords from a sentence and accepts the following arguments:
optional arguments:
  -h, --help         show this help message and exit
  --sentence SEN        sentence to extract keywords
  --path LOAD        path to load model from


python --sentence "BERT is a great model." --path ""           


You can also train it from scratch using BERT's pre-trained model. The
script can be utilized for training and accepts the following arguments:
optional arguments:
  -h, --help         show this help message and exit
  --data DATA        location of the data corpus
  --lr LR            initial learning rate
  --epochs EPOCHS    upper epoch limit
  --batch_size N     batch size
  --seq_len N        sequence length
  --save SAVE        path to save the final model


python --data "maui-semeval2010-train" --lr 2e-5 --batch_size 32 --save "" --epochs 3      

This model has been trained on SemEval 2010 dataset (scientific publications). You can swap this with your own custom dataset.

Code explanations

I have provided the explanation of keyphrase extraction in the form of python notebook which you can view here

Hyper-parameter Tuning

I ran ablation experiments according to the BERT paper and these are the results. I suggest to use parameters in line 4. All training was done on batch size of 32.

| Learning Rate | Number of Epochs | Validation loss | Validation Accuracy | F1-Score | |--------------- |------------------ |----------------- |--------------------- |-------------- | | 3.00E-05 | 3 | 0.05294724515 | 98.30% | 0.5318559557 | | 5.00E-05 | 3 | 0.04899719357 | 98.47% | 0.56218628 | | 2.00E-05 | 3 | 0.05733459462 | 98.15% | 0.4390547264 | | 3.00E-05 | 4 | 0.05020467712 | 98.48% | 0.5528169014 | | 5.00E-05 | 4 | 0.05194576555 | 98.43% | 0.5780836421 | | 2.00E-05 | 4 | 0.05373481681 | 98.25% | 0.5019740553 |

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.