Sai5480's picture
Add config.json for mal tokenizer
818c43f verified
{
"model_type": "sentencepiece",
"tokenizer_class": "PreTrainedTokenizerFast",
"vocab_size": 128000,
"model_max_length": 512,
"unk_token": "<unk>",
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "<pad>"
}