DeepPavlov

by deepmipt

deepmipt / DeepPavlov

An open source library for deep learning end-to-end dialog systems and chatbots.

4.7K Stars 854 Forks Last release: about 2 months ago (0.12.1) Apache License 2.0 2.6K Commits 39 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

License Apache 2.0 Python 3.6, 3.7 Downloads

DeepPavlov is an open-source conversational AI library built on TensorFlow, Keras and PyTorch.

DeepPavlov is designed for * development of production ready chat-bots and complex conversational systems, * research in the area of NLP and, particularly, of dialog systems.

Quick Links

Please leave us your feedback on how we can improve the DeepPavlov framework.

Models

Named Entity Recognition | Slot filling

Intent/Sentence Classification | Question Answering over Text (SQuAD)

Knowledge Base Question Answering

Sentence Similarity/Ranking | TF-IDF Ranking

Morphological tagging | Syntactic parsing

Automatic Spelling Correction | ELMo training and fine-tuning

Speech recognition and synthesis (ASR and TTS) based on NVIDIA NeMo

Entity Linking | Multitask BERT

Skills

Goal(Task)-oriented Bot | Seq2seq Goal-Oriented bot

Open Domain Questions Answering | eCommerce Bot

Frequently Asked Questions Answering | Pattern Matching

Embeddings

BERT embeddings for the Russian, Polish, Bulgarian, Czech, and informal English

ELMo embeddings for the Russian language

FastText embeddings for the Russian language

Auto ML

Tuning Models with Evolutionary Algorithm

Integrations

REST API | Socket API | Yandex Alice

Telegram | Microsoft Bot Framework

Amazon Alexa | Amazon AWS

Installation

  1. We support

    Linux
    and
    Windows
    platforms,
    Python 3.6
    and
    Python 3.7
    • Python 3.5
      is not supported!
    • installation for
      Windows
      requires
      Git
      (for example, git) and
      Visual Studio 2015/2017
      with
      C++
      build tools installed!
  2. Create and activate a virtual environment:

    • Linux
      python -m venv env
      source ./env/bin/activate
      
    • Windows
      python -m venv env
      .\env\Scripts\activate.bat
      
  3. Install the package inside the environment:

    pip install deeppavlov
    

QuickStart

There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is determined by its config file.

List of models is available on the doc page in the

deeppavlov.configs
(Python):
from deeppavlov import configs

When you're decided on the model (+ config file), there are two ways to train, evaluate and infer it:

GPU requirements

To run supported DeepPavlov models on GPU you should have CUDA 10.0 installed on your host machine and TensorFlow with GPU support (

tensorflow-gpu
) installed in your python environment. Current supported TensorFlow version is 1.15.2. Run
pip install tensorflow-gpu==1.15.2

before installing model's package requirements to install supported

tensorflow-gpu
version.

Before making choice of an interface, install model's package requirements (CLI):

python -m deeppavlov install 
  • where
     is path to the chosen model's config file (e.g.
    
    deeppavlov/configs/ner/slotfill_dstc2.json
    ) or just name without .json extension (e.g.
    slotfill_dstc2
    )

Command line interface (CLI)

To get predictions from a model interactively through CLI, run

python -m deeppavlov interact  [-d]
  • -d
    downloads required data -- pretrained model files and embeddings (optional).

You can train it in the same simple way:

python -m deeppavlov train  [-d]

Dataset will be downloaded regardless of whether there was

-d
flag or not.

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

There are even more actions you can perform with configs:

python -m deeppavlov   [-d]
  •  can be
    
    
    • download
      to download model's data (same as
      -d
      ),
    • train
      to train the model on the data specified in the config file,
    • evaluate
      to calculate metrics on the same dataset,
    • interact
      to interact via CLI,
    • riseapi
      to run a REST API server (see doc),
    • telegram
      to run as a Telegram bot (see doc),
    • msbot
      to run a Miscrosoft Bot Framework server (see doc),
    • predict
      to get prediction for samples from stdin or from if
      -f 
      is specified.
  •  specifies path (or name) of model's config file
  • -d
    downloads required data

Python

To get predictions from a model interactively through Python, run

from deeppavlov import build_model

model = build_model(, download=True)

get predictions for 'input_text1', 'input_text2'

model(['input_text1', 'input_text2'])

  • where
    download=True
    downloads required data from web -- pretrained model files and embeddings (optional),
  •  is path to the chosen model's config file (e.g.
    
    "deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"
    ) or
    deeppavlov.configs
    attribute (e.g.
    deeppavlov.configs.ner.ner_ontonotes_bert_mult
    without quotation marks).

You can train it in the same simple way:

from deeppavlov import train_model 

model = train_model(, download=True)

  • download=True
    downloads pretrained model, therefore the pretrained model will be, first, loaded and then train (optional).

Dataset will be downloaded regardless of whether there was

-d
flag or not.

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

You can also calculate metrics on the dataset specified in your config file:

from deeppavlov import evaluate_model 

model = evaluate_model(, download=True)

There are also available integrations with various messengers, see Telegram Bot doc page and others in the Integrations section for more info.

Breaking Changes

Breaking changes in version 0.7.0 - in dialog logger config file dialogloggerconfig.json

agent_name
parameter was renamed to
logger_name
, the default value was changed - Agent, Skill, eCommerce Bot and Pattern Matching classes were moved to deeppavlov.deprecated - AIML Skill, RASA Skill, Yandex Alice, Amazon Alexa, Microsoft Bot Framework and Telegram integration interfaces were changed -
/start
and
/help
Telegram messages were moved from
models_info.json
to server_config.json - risesocket request and response format was changed - riseapi and risesocket model-specific properties parametrization was changed

Breaking changes in version 0.6.0 - REST API: - all models default endpoints were renamed to

/model
- by default model arguments names are taken from
chainer.in
configuration parameter instead of pre-set names from a settings file - swagger api endpoint moved from
/apidocs
to
/docs
- when using
"max_proba": true
in a
proba2labels
component
for classification, it will return single label for every batch element instead of a list. One can set
"top_n": 1
to get batches of single item lists as before

Breaking changes in version 0.5.0 - dependencies have to be reinstalled for most pipeline configurations - models depending on

tensorflow
require
CUDA 10.0
to run on GPU instead of
CUDA 9.0
- scikit-learn models have to be redownloaded or retrained

Breaking changes in version 0.4.0! - default target variable name for neural evolution was changed from

MODELS_PATH
to
MODEL_PATH
.

Breaking changes in version 0.3.0! - component option

fit_on_batch
in configuration files was removed and replaced with adaptive usage of the
fit_on
parameter.

Breaking changes in version 0.2.0! -

utils
module was moved from repository root in to
deeppavlov
module -
ms_bot_framework_utils
,
server_utils
,
telegram utils
modules was renamed to
ms_bot_framework
,
server
and
telegram
correspondingly - rename metric functions
exact_match
to
squad_v2_em
and
squad_f1
to
squad_v2_f1
- replace dashes in configs name with underscores

Breaking changes in version 0.1.0! - As of

version 0.1.0
all models, embeddings and other downloaded data for provided configurations are by default downloaded to the
.deeppavlov
directory in current user's home directory. This can be changed on per-model basis by modifying a
ROOT_PATH
variable or related fields one by one in model's configuration file.
  • In configuration files, for all features/models, dataset readers and iterators

    "name"
    and
    "class"
    fields are combined into the
    "class_name"
    field.
  • deeppavlov.core.commands.infer.build_model_from_config()
    was renamed to
    build_model
    and can be imported from the
    deeppavlov
    module directly.
  • The way arguments are passed to metrics functions during training and evaluation was changed and documented.

License

DeepPavlov is Apache 2.0 - licensed.

The Team

DeepPavlov is built and maintained by Neural Networks and Deep Learning Lab at MIPT.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.