Sai5480's picture
Add config.json for guj tokenizer
2942126 verified
raw
history blame
227 Bytes
{
"model_type": "sentencepiece",
"tokenizer_class": "PreTrainedTokenizerFast",
"vocab_size": 128000,
"model_max_length": 512,
"unk_token": "<unk>",
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "<pad>"
}