|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- it |
|
|
tags: |
|
|
- fill-mask |
|
|
- masked-lm |
|
|
- long-context |
|
|
- modernbert |
|
|
- italian |
|
|
pipeline_tag: fill-mask |
|
|
--- |
|
|
|
|
|
# Italian ModernBERT |
|
|
|
|
|
--- |
|
|
|
|
|
**💡 Found this resource helpful?** Creating and maintaining open source AI models and datasets requires significant computational resources. If this work has been valuable to you, consider [supporting my research](https://buymeacoffee.com/michele.montebovi) to help me continue building tools that benefit the entire AI community. Every contribution directly funds more open source innovation! ☕ |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Description |
|
|
|
|
|
Italian ModernBERT (DeepMount00/Italian-ModernBERT-base) is a specialized Italian language version of ModernBERT, specifically pre-trained on Italian text corpora. This model is designed exclusively for Italian language tasks. |
|
|
|
|
|
## Key Features |
|
|
|
|
|
- **Architecture**: Based on ModernBERT-base (22 layers, 149M parameters) |
|
|
- **Context Length**: 8,192 tokens |
|
|
- **Language**: Italian-only |
|
|
- **Tokenizer**: Custom tokenizer optimized for Italian language |
|
|
- **Training**: Pre-trained on Italian text corpus |
|
|
|
|
|
## Technical Details |
|
|
|
|
|
- Uses Rotary Positional Embeddings (RoPE) |
|
|
- Implements Local-Global Alternating Attention |
|
|
- Supports Flash Attention 2 for optimal performance |
|
|
- No token type IDs required |
|
|
|
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Optimized only for Italian language processing |
|
|
- Not suitable for other languages |
|
|
- May reflect biases present in training data |
|
|
|
