devngho
/

phi3-jamo-ko-large-tokenizer

Model card Files Files and versions

phi3-jamo-ko-large-tokenizer / README.md

devngho's picture

Update README.md

ec360a4 verified 12 months ago

|

history blame contribute delete

227 Bytes

metadata

library_name: transformers
tags: []

expanded 49556 tokens

Following The Optimal Vocabulary Size Predictor, I recommend using this tokenizer with 13B model.