File size: 323 Bytes
ddb7b9a 2df511e 8d12b54 2df511e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
---
license: apache-2.0
---
The Hugging Face fast tokenizer for LLM-jp ABCI challenge 2023.
The vocab size is 96,867.
Requirements:
- transformers>=4.34.0
- tokenizers>=0.14.0
- torch
Usage:
```Python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v22b2")
``` |