HuBERTUSz
Collection
Small Hungarian BERT models
•
7 items
•
Updated
hubert-tiny-wiki
This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks.
Pre-Training Parameters:
First phase:
- Training steps: 500.000
- Sequence length: 128
- Batch size: 1024
Second phase:
- Training steps: 100.000
- Sequence length: 512
- Batch size: 384
Framework versions
- Transformers 4.21.3
- TensorFlow 2.10.0
- Datasets 2.4.0
- Tokenizers 0.12.1
Acknowledgement
- Downloads last month
- 1
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support