Sai5480
/

monolingual-tokenizer-native-tam-vocab-128000

Model card Files Files and versions

Monolingual Tokenizer - Tamil (Vocab 128000)

This is a monolingual tokenizer trained on Tamil text with vocabulary size 128000.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("monolingual-tokenizer-native-tam-vocab-128000")

Files

tam.model: SentencePiece model file
tam.vocab: Vocabulary file
config.json: Tokenizer configuration

Training Details

Language: Tamil (tam)
Vocabulary Size: 128000
Model Type: SentencePiece Unigram

Downloads last month: 8

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support