Need help with HanLP?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

hankcs
22.8K Stars 6.1K Forks Apache License 2.0 1.5K Commits 5 Opened issues

Description

Natural Language Processing for the next decade. Tokenization, Part-of-Speech Tagging, Named Entity Recognition, Syntactic & Semantic Dependency Parsing, Document Classification

Services available

!
?

Need anything else?

Contributors list

# 103
Java
Python
Tensorf...
text-cl...
1294 commits
# 6,752
C#
sony
qq
pos-tag...
13 commits
# 10,761
depende...
Python
pos-tag...
Java
11 commits
# 21,632
depende...
Python
pos-tag...
Java
4 commits
# 21,633
depende...
Python
pos-tag...
Java
4 commits
# 19,641
Python
TeX
Shell
depende...
4 commits
# 25,580
Python
depende...
HTML
Jupyter...
3 commits
# 25,781
Go
Shell
depende...
Python
3 commits
# 26,115
depende...
Python
pos-tag...
Shell
3 commits
# 26,135
depende...
Python
pos-tag...
Java
3 commits
# 26,133
depende...
Python
pos-tag...
Java
3 commits
# 32,528
depende...
Python
pos-tag...
Shell
2 commits
# 33,735
depende...
Python
pos-tag...
Java
2 commits
# 33,733
depende...
Python
pos-tag...
Java
2 commits
# 7,833
HTML
Python
Jupyter...
Tensorf...
2 commits
# 33,734
depende...
Python
pos-tag...
Java
2 commits
# 31,207
burpsui...
ecmascr...
pos-tag...
Redis
2 commits
# 6,182
Python
cpluspl...
synonym...
Tensorf...
2 commits
# 33,727
depende...
Python
pos-tag...
Java
2 commits
# 50,997
depende...
Python
pos-tag...
Java
1 commit

HanLP: Han Language Processing

中文 | docs | 1.x | forum | Open In Colab

The multilingual NLP library for researchers and companies, built on PyTorch and TensorFlow 2.x, for advancing state-of-the-art deep learning techniques in both academia and industry. HanLP was designed from day one to be efficient, user friendly and extendable.

Thanks to open-access corpora like Universal Dependencies and OntoNotes, HanLP 2.1 now offers 10 joint tasks on 104 languages: tokenization, lemmatization, part-of-speech tagging, token feature extraction, dependency parsing, constituency parsing, semantic role labeling, semantic dependency parsing, abstract meaning representation (AMR) parsing.

For end users, HanLP offers light-weighted RESTful APIs and native Python APIs.

RESTful APIs

Tiny packages in several KBs for agile development and mobile applications. Although anonymous users are welcomed, an auth key is suggested and a free one can be applied here under the CC BY-NC-SA 4.0 license.

### Python

pip install hanlp_restful

Create a client with our API endpoint and your auth.

from hanlp_restful import HanLPClient
HanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None, language='mul')

Java

Insert the following dependency into your

pom.xml
.
  com.hankcs.hanlp.restful
  hanlp-restful
  0.0.4

Create a client with our API endpoint and your auth.

HanLPClient HanLP = new HanLPClient("https://hanlp.hankcs.com/api", null, "mul");

Quick Start

No matter which language you use, the same interface can be used to parse a document.

HanLP.parse("In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments. 2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。")

See docs for visualization, annotation guidelines and more details.

Native APIs

pip install hanlp

HanLP requires Python 3.6 or later. GPU/TPU is suggested but not mandatory.

Quick Start

import hanlp
HanLP = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)
print(HanLP(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',
             '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',
             '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。']))

In particular, the Python

HanLPClient
can also be used as a callable function following the same semantics. See docs for visualization, annotation guidelines and more details.

Train Your Own Models

To write DL models is not hard, the real hard thing is to write a model able to reproduce the scores in papers. The snippet below shows how to surpass the state-of-the-art tokenizer in 6 minutes.

tokenizer = TransformerTaggingTokenizer()
save_dir = 'data/model/cws/sighan2005_pku_bert_base_96.70'
tokenizer.fit(
    SIGHAN2005_PKU_TRAIN_ALL,
    SIGHAN2005_PKU_TEST,  # Conventionally, no devset is used. See Tian et al. (2020).
    save_dir,
    'bert-base-chinese',
    max_seq_len=300,
    char_level=True,
    hard_constraint=True,
    sampler_builder=SortingSamplerBuilder(batch_size=32),
    epochs=3,
    adam_epsilon=1e-6,
    warmup_steps=0.1,
    weight_decay=0.01,
    word_dropout=0.1,
    seed=1609836303,
)
tokenizer.evaluate(SIGHAN2005_PKU_TEST, save_dir)

The result is guaranteed to be

96.70
as the random feed is fixed. Different from some overclaiming papers and projects, HanLP promises every single digit in our scores is reproducible. Any issues on reproducibility will be treated and solved as a top-priority fatal bug.

Performance

lang corpora model tok pos ner dep con srl sdp lem fea amr
fine coarse ctb pku 863 ud pku msra ontonotes SemEval16 DM PAS PSD
mul UD2.7
OntoNotes5
small 98.62 - - - - 93.23 - - 74.42 79.10 76.85 70.63 - 91.19 93.67 85.34 87.71 84.51 -
base 99.67 - - - - 96.51 - - 80.76 87.64 80.58 77.22 - 94.38 96.10 86.64 94.37 91.60 -
zh open small 97.25 - 96.66 - - - - - 95.00 84.57 87.62 73.40 84.57 - - - - - -
base 97.50 - 97.07 - - - - - 96.04 87.11 89.84 77.78 87.11 - - - - - -
close small 96.70 95.93 96.87 97.56 95.05 - 96.22 95.74 76.79 84.44 88.13 75.81 74.28 - - - - - -
base 97.52 96.44 96.99 97.59 95.29 - 96.48 95.72 77.77 85.29 88.57 76.52 73.76 - - - - - -
  • AMR models will be released once our paper gets accepted.

Citing

If you use HanLP in your research, please cite this repository.

@software{hanlp2,
  author = {Han He},
  title = {{HanLP: Han Language Processing}},
  year = {2020},
  url = {https://github.com/hankcs/HanLP},
}

License

Codes

HanLP is licensed under Apache License 2.0. You can use HanLP in your commercial products for free. We would appreciate it if you add a link to HanLP on your website.

Models

Unless otherwise specified, all models in HanLP are licensed under CC BY-NC-SA 4.0.

References

https://hanlp.hankcs.com/docs/references.html

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.