| license: apache-2.0 | |
| The Hugging Face fast tokenizer for LLM-jp ABCI challenge 2023. | |
| The vocab size is 96,867. | |
| Requirements: | |
| - transformers>=4.34.0 | |
| - tokenizers>=0.14.0 | |
| - torch | |
| Usage: | |
| ```Python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v22b2") | |
| ``` |