Need help with sensegram?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

uhh-lt
214 Stars 49 Forks 283 Commits 12 Opened issues

Description

Making sense embedding out of word embeddings using graph-based word sense induction

Services available

!
?

Need anything else?

Contributors list

SenseGram

This repository contains implementation of a method that takes as an input a word embeddings, such as word2vec and splits different senses of the input words. For instance, the vector for the word "table" will be split into "table (data)" and "table (furniture)" as shown below.

Our method performs word sense induction and disambiguation based on sense embeddings. Sense inventory is induced from exhisting word embeddings via clustering of ego-networks of related words. Detailed description of the method is available in the original paper:

The picture below illustrates the main idea of the underlying approach:

ego

If you use the method please cite the following paper:

@InProceedings{pelevina-EtAl:2016:RepL4NLP,
  author    = {Pelevina, Maria  and  Arefiev, Nikolay  and  Biemann, Chris  and  Panchenko, Alexander},
  title     = {Making Sense of Word Embeddings},
  booktitle = {Proceedings of the 1st Workshop on Representation Learning for NLP},
  month     = {August},
  year      = {2016},
  address   = {Berlin, Germany},
  publisher = {Association for Computational Linguistics},
  pages     = {174--183},
  url       = {http://anthology.aclweb.org/W16-1620}
}

Use cases

This software can be used to:

  • Generation of word sense embeddigns from a raw text corpus

  • Generation of word sense embeddings from a pretrained word embeddings (in the word2vec format)

  • Generation of graphs of semantically related words

  • Generation of graphs of semantically related word senses

  • Generation of a word sense inventory specific to the input text corpus

Installation

This project is implemented in Python 3. It makes use of the word2vec toolkit (via gensim), FAISS for computation of graphs of related words, and the Chinese Whispers graph clustering algorithm. We suggest using Ubuntu Linux 16.04 for computation of the models and using it on a computational server (ideally from 64Gb of RAM and 16 cores) as some stages are computational intensive. To install all dependencies on Ubuntu Linux 16.04 use the following commands:

git clone --recursive https://github.com/tudarmstadt-lt/sensegram.git
make install-ubuntu-16-04

Optional: Set the

PYTHONPATH
variable to the root directory of this repository (needed only for working with the "egvi" scripts), e.g. ``export PYTHONPATH="/home/user/sensegram:$PYTHONPATH"

Note that this command also will bring you an appropriate vesion of Python 3 via Anaconda. If you already have a properly configured recent version of Python 3 and/or running a system different from Ubuntu 16.04, use the command

make install
to install the dependencies. Note however, that in this case, you will also need to install manually binary dependencies required by FAISS yourself.

Training a new model from a text corpus

The way to train your own sense embeddings is with the

train.py
script. You will have to provide a raw text corpus as input. If you run
train.py
with no parameters, it will print usage information:
usage: train.py [-h] [-cbow CBOW] [-size SIZE] [-window WINDOW]
                [-threads THREADS] [-iter ITER] [-min_count MIN_COUNT] [-N N]
                [-n N] [-min_size MIN_SIZE] [-make-pcz]
                train_corpus

Performs training of a word sense embeddings model from a raw text corpus using the SkipGram approach based on word2vec and graph clustering of ego networks of semantically related terms.

positional arguments: train_corpus Path to a training corpus in text form (can be .gz).

optional arguments: -h, --help show this help message and exit -cbow CBOW Use the continuous bag of words model (default is 1, use 0 for the skip-gram model). -size SIZE Set size of word vectors (default is 300). -window WINDOW Set max skip length between words (default is 5). -threads THREADS Use threads (default 40). -iter ITER Run training iterations (default 5). -min_count MIN_COUNT This will discard words that appear less than times (default is 10). -N N Number of nodes in each ego-network (default is 200). -n N Maximum number of edges a node can have in the network (default is 200). -min_size MIN_SIZE Minimum size of the cluster (default is 5). -make-pcz Perform two extra steps to label the original sense inventory with hypernymy labels and disambiguate the list of related words.The obtained resource is called proto-concepualization or PCZ.

The training produces following output files:

  • model/ + CORPUS_NAME + .word_vectors
    - word vectors in the word2vec text format
  • model/ + CORPUS_NAME + .sense_vectors
    - sense vectors in the word2vec text format
  • model/ + CORPUS_NAME + .sense_vectors.inventory.csv
    - sense probabilities in TSV format

In addition, it produces several intermediary files that can be investigated for error analysis or removed after training:

  • model/ + CORPUS_NAME + .graph
    - word similarity graph (distributional thesaurus) in TSV format
  • model/ + corpus_name + .clusters
    - sense clusters produced by chinese-whispers in TSV format
  • model/ + corpus_name + .minsize + MIN_SIZE
    - clusters that remained after filtering out of small clusters in TSV format

In train.sh we provide an example for usage of the

train.py
script. You can test it using the command
make train
. More useful commands can be found in the Makefile.

Using a pre-trained model

See the QuickStart tutorial on how to perform word sense disambiguation and inspection of a trained model.

You can downlooad pre-trained models for English, German, and Russian. Note that to run examples from the QuickStart you only need files with extensions

.word_vectors
,
.sense_vectors
, and
.sense_vectors.inventory.csv
. Other files are supplementary.

Transforming pre-trained word embeddings to sense embeddings

Instead of learning a model from a text corpus, you can provide a pre-trained word embedding model. To do so, you just neeed to:

  1. Save the word embeddings file (in word2vec text format) with the extension

    .word_vectors
    , e.g.
    wikipedia.word_vectors
    .
  2. Run the

    train.py
    script inducating the path to the word embeddings file, e.g.:

python train.py model/wikipedia

Note: do not indicate the

.word_vectors
extension when launching the train.py script.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.