Julia Implementation of Transformer models
Julia implementation of transformer-based models, with Flux.jl.
In the Julia REPL:
]add Transformers
For using GPU, install & build:
]add CUDA]build
julia> using CUDA
julia> using Transformers
#run the model below . . .
Using pretrained Bert with
Transformers.jl.
using Transformers using Transformers.Basic using Transformers.PretrainENV["DATADEPS_ALWAYS_ACCEPT"] = true
bert_model, wordpiece, tokenizer = pretrain"bert-uncased_L-12_H-768_A-12" vocab = Vocabulary(wordpiece)
text1 = "Peter Piper picked a peck of pickled peppers" |> tokenizer |> wordpiece text2 = "Fuzzy Wuzzy was a bear" |> tokenizer |> wordpiece
text = ["[CLS]"; text1; "[SEP]"; text2; "[SEP]"] @assert text == [ "[CLS]", "peter", "piper", "picked", "a", "peck", "of", "pick", "##led", "peppers", "[SEP]", "fuzzy", "wu", "##zzy", "was", "a", "bear", "[SEP]" ]
token_indices = vocab(text) segment_indices = [fill(1, length(text1)+2); fill(2, length(text2)+1)]
sample = (tok = token_indices, segment = segment_indices)
bert_embedding = sample |> bert_model.embed feature_tensors = bert_embedding |> bert_model.transformers
See
examplefolder for the complete example.
We have some support for the models from
huggingface/transformers.
using Transformers.HuggingFaceloading a model from huggingface model hub
julia> model = hgf"bert-base-cased:forquestionanswering"; ┌ Warning: Transformers.HuggingFace.HGFBertForQuestionAnswering doesn't have field cls. └ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:46 ┌ Warning: Some fields of Transformers.HuggingFace.HGFBertForQuestionAnswering aren't initialized with loaded state: qa_outputs └ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:52
Current we only support a few model and the tokenizer part is not finished yet.
If you want to know more about this package, see the document and the series of blog posts I wrote for JSoC and GSoC. You can also tag me (@chengchingwen) on Julia's slack or discourse if you have any questions, or just create a new Issue on GitHub.
Transformerand
TransformerDecodersupport for both 2d & 3d data.
PositionEmbeddingimplementation.
Positionwisefor handling 2d & 3d input.
examplefolder)
Transformers.HuggingFacefor handling pretrains from
huggingface/transformers