Sai5480/monolingual-tokenizer-native-urd-vocab-128000

Monolingual Tokenizer - Urdu (Vocab 128000)

This is a monolingual tokenizer trained on Urdu text with vocabulary size 128000.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("monolingual-tokenizer-native-urd-vocab-128000")

Files

• urd.model: SentencePiece model file
• urd.vocab: Vocabulary file
• config.json: Tokenizer configuration

Training Details

• Language: Urdu (urd)
• Vocabulary Size: 128000
• Model Type: SentencePiece Unigram