A Curated List of Dataset and Usable Library Resources for NLP in Bahasa Indonesia
This repository provides link to useful dataset and another resources for NLP in Bahasa Indonesia.
Last Update: 24 Des 2020
1) (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negatifta2.txt 2) (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negativeadd.txt 3) (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negativekeyword.txt 4) (Negative) https://github.com/masdevid/ID-OpinionWords/blob/master/negative.txt 5) (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positifta2.txt 6) (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positiveadd.txt 7) (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positivekeyword.txt 8) (Positive) https://github.com/masdevid/ID-OpinionWords/blob/master/positive.txt 9) (Score) https://github.com/agusmakmun/SentiStrengthID/blob/master/iddict/sentimentword.txt 10) (InSet Lexicon) https://github.com/fajri91/InSet [Paper] 11) (Twitter Labelled Sentiment) https://www.researchgate.net/profile/RidiFerdiana/publication/339936724IndonesianSentimentTwitterDataset/data/5e6d64c6a6fdccf994ca18aa/Indonesian-Sentiment-Twitter-Dataset.zip?origin=publicationDetail_linkedData [Paper]
1) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/psuf.txt 2) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/lldr.txt 3) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/opos.txt 4) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/ptit.txt
1) https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/rootword.txt 2) https://github.com/sastrawi/sastrawi/blob/master/data/kata-dasar.original.txt 3) https://github.com/sastrawi/sastrawi/blob/master/data/kata-dasar.txt 4) https://github.com/prasastoadi/serangkai/blob/master/serangkai/kamus/data/kamus-kata-dasar.csv
I have made the combined root words list from all of the above repositories.
1) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/kbba.txt 2) https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/slangword.txt 3) https://github.com/panggi/pujangga/blob/master/resource/formalization/formalizationDict.txt
I have made the combined slang words dictionary from all of the above repositories.
1) https://github.com/yasirutomo/python-sentianalysis-id/blob/master/data/feature_list/stopwordsID.txt 2) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/stopword.txt 3) https://github.com/abhimantramb/elang/tree/master/word2vec/utils/stopwords-list
I have made the combined stop words list from all of the above repositories.
1) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/emoticon.txt 2) https://github.com/jolicode/emoji-search/blob/master/synonyms/cldr-emoji-annotation-synonyms-id.txt 3) https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/emoticon.txt
1) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/acronym.txt 2) https://github.com/panggi/pujangga/blob/master/resource/sentencedetector/acronym.txt 3) https://id.wiktionary.org/wiki/Lampiran:DaftarsingkatandanakronimdalambahasaIndonesia#A
1) https://github.com/abhimantramb/elang/blob/master/word2vec/utils/indonesian-region.txt 2) https://github.com/edwardsamuel/Wilayah-Administratif-Indonesia/tree/master/csv 3) https://github.com/pentagonal/Indonesia-Postal-Code/tree/master/Csv
1) https://github.com/abhimantramb/elang/blob/master/word2vec/utils/swear-words.txt
1) https://github.com/panggi/pujangga/blob/master/resource/tokenizer/compositewords.txt
1) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/country.txt
1) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/lpre.txt
1) https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/ppre.txt
1) https://github.com/seuriously/genderprediction/blob/master/namatraining.txt
1) https://github.com/panggi/pujangga/blob/master/resource/reference/opre.txt
1) https://github.com/panggi/pujangga/blob/master/resource/netagger/morphologicalfeature/number.txt
1) https://github.com/onlyphantom/elang/blob/master/build/lib/elang/word2vec/utils/negative/calendar-words.txt
1) Product NER. https://github.com/dziem/proner-labeled-text 2) NER-grit. https://github.com/grit-id/nergrit-corpus
1) https://github.com/famrashel/idn-tagged-corpus 2) https://github.com/kmkurn/id-pos-tagging/blob/master/data/dataset.tar.gz
1) https://github.com/google-research-datasets/tydiqa
1) https://github.com/okkyibrohim/id-multi-label-hate-speech-and-abusive-language-detection
1) https://github.com/kata-ai/indosum
1) https://github.com/kata-ai/kawat
1) OSCAR. https://oscar-corpus.com/ 2) Online Newspaper. https://github.com/feryandi/Dataset-Artikel
1) https://github.com/ilhamfp/puisi-pantun-generator
1) https://medium.com/@puspitakaban/pos-tagging-bahasa-indonesia-dengan-flair-nlp-c12e45542860 2) Manually Tagged Indonesian Corpus [Paper] [GitHub]
1) Indo-BERT. https://github.com/indobenchmark/indonlu & https://huggingface.co/indobenchmark/indobert-base-p1 1) Transformer-based Pre-trained Model in Bahasa. https://github.com/cahya-wirawan/indonesian-language-models/tree/master/Transformers 1) Generate Word-Embedding / Sentence-Embedding using pre-Trained Multilingual Bert model. (https://colab.research.google.com/drive/1yFphU6PW9Uo6lmDly_ud9a6c4RCYlwdX#scrollTo=Zn0n2S-FWZih). P.S: Just change the model using 'bert-base-multilingual-uncased' 2) https://github.com/meisaputri21/Indonesian-Twitter-Emotion-Dataset. [Paper] 3) https://github.com/Kyubyong/wordvectors 4) https://drive.google.com/uc?id=0B5YTktu2dOKKNUY1OWJORlZTcUU&export=download 5) https://github.com/deryrahman/word2vec-bahasa-indonesia 6) https://sites.google.com/site/rmyeid/projects/polyglot
1) (FastText). https://structilmy.com/2019/08/membuat-model-word-embedding-fasttext-bahasa-indonesia/ 2) (Word2Vec). https://yudiwbs.wordpress.com/2018/03/31/word2vec-wikipedia-bahasa-indonesia-dengan-python-gensim/
1) Pujangga: Indonesian Natural Language Processing REST API. https://github.com/panggi/pujangga 2) Sastrawi Stemmer Bahasa Indonesia. https://github.com/sastrawi/sastrawi 3) NLP-ID. https://github.com/kumparan/nlp-id 4) MorphInd: Indonesian Morphological Analyzer. http://septinalarasati.com/morphind/ 5) INDRA: Indonesian Resource Grammar. https://github.com/davidmoeljadi/INDRA 6) Typo Checker. https://github.com/mamat-rahmat/checker_id 7) Multilingual NLP Package. https://github.com/flairNLP/flair 9) spaCy [GitHub] [Tutorial] 9) https://github.com/yohanesgultom/nlp-experiments 10) https://github.com/yasirutomo/python-sentianalysis-id 11) https://github.com/riochr17/Analisis-Sentimen-ID 12) https://github.com/yusufsyaifudin/indonesia-ner
Sometimes there is an english word within our text and we have to translate it. We can exploit the english word dictionary provided here and we can use the Google Translate API for Python
You can adjust this code with Bahasa corpus to do the spelling correction
1) (Introduction to LSA & LDA). https://monkeylearn.com/blog/introduction-to-topic-modeling/
2) (Introduction to LDA w/ Code & Tips). https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/
3) (Topic Modeling Methods Comparison Paper). https://thesai.org/Downloads/Volume6No1/Paper21-ASurveyofTopicModelinginTextMining.pdf
4) (Original LDA Paper). http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
5) (LDA Python Library). https://pypi.org/project/lda/; https://radimrehurek.com/gensim/models/ldamodel.html; https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html
6) (Original CTM Paper). http://people.ee.duke.edu/~lcarin/Blei2005CTM.pdf
7) (CTM Python Library). https://pypi.org/project/tomotopy/; https://github.com/kzhai/PyCTM
8) (Gaussian LDA Paper). https://www.aclweb.org/anthology/P15-1077.pdf
9) (Gaussian LDA Library). https://github.com/rajarshd/GaussianLDA
10) (Temporal Topic Modeling Comparison Paper). https://thesai.org/Downloads/Volume6No1/Paper21-ASurveyofTopicModelinginTextMining.pdf
11) (TOT: A Non-Markov Continuous-Time Model of Topical Trends Paper). https://people.cs.umass.edu/~mccallum/papers/tot-kdd06s.pdf
12) (TOT Library). https://github.com/ahmaurya/topicsover_time
13) (Example of LDA in Bahasa Project Code). https://github.com/kirralabs/text-clustering
1) (Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach) https://arxiv.org/pdf/1909.00161.pdf | https://github.com/yinwenpeng/BenchmarkingZeroShot 2) (Integrating Semantic Knowledge to Tackle Zero-shot Text Classification) https://arxiv.org/abs/1903.12626 | https://github.com/JingqingZ/KG4ZeroShotText 3) (Train Once, Test Anywhere: Zero-Shot Learning for Text Classification) https://arxiv.org/abs/1712.05972 | https://amitness.com/2020/05/zero-shot-text-classification/ 4) (Zero-shot Text Classification With Generative Language Models) https://arxiv.org/abs/1912.10165 | https://amitness.com/2020/06/zero-shot-classification-via-generation/ 5) (Zero-shot User Intent Detection via Capsule Neural Networks) https://arxiv.org/abs/1809.00385 | https://github.com/congyingxia/ZeroShotCapsule
1) (Few-shot Text Classification with Distributional Signatures) https://arxiv.org/pdf/1908.06039.pdf | https://github.com/YujiaBao/Distributional-Signatures 2) (Few Shot Text Classification with a Human in the Loop) https://katbailey.github.io/talks/Few-shot%20text%20classification.pdf | https://github.com/katbailey/few-shot-text-classification 3) (Induction Networks for Few-Shot Text Classification) https://arxiv.org/pdf/1902.10482v2.pdf | https://github.com/zhongyuchen/few-shot-learning
1) GetOldTweets3. https://github.com/Mottl/GetOldTweets3
Usage:
bash import GetOldTweets3 as got tweetCriteria=got.manager.TweetCriteria().setQuerySearch('#CoronaVirusIndonesia').setSince("2020-01-01").setUntil("2020-03-05").setNear("Jakarta, Indonesia").setLang("id") tweets=got.manager.TweetManager.getTweets(tweetCriteria) for tweet in tweets: print(tweet.username) print(tweet.text) print(tweet.date) print("tweet.to") print("tweet.retweets") print("tweet.favorites") print("tweet.mentions") print("tweet.hashtags") print("tweet.geo")
2) Tweepy. http://docs.tweepy.org/en/latest/
Step-by-step how to use Tweepy. https://towardsdatascience.com/how-to-scrape-tweets-from-twitter-59287e20f0f1
Sign in to Twitter Developer. https://developer.twitter.com/en
Full List of Tweets Object. https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/tweet-object
Increasing Tweepy’s standard API search limit. https://bhaskarvk.github.io/2015/01/how-to-use-twitters-search-rest-api-most-effectively./
1) https://github.com/irfnrdh/Awesome-Indonesia-NLP 2) https://github.com/kirralabs/indonesian-NLP-resources