|
|
---
|
|
|
license: mit
|
|
|
language:
|
|
|
- en
|
|
|
pipeline_tag: text-generation
|
|
|
tags:
|
|
|
- phi
|
|
|
- nlp
|
|
|
- math
|
|
|
- code
|
|
|
- chat
|
|
|
- conversational
|
|
|
inference:
|
|
|
parameters:
|
|
|
temperature: 0
|
|
|
widget:
|
|
|
- messages:
|
|
|
- role: user
|
|
|
content: How should I explain the Internet?
|
|
|
library_name: transformers
|
|
|
---
|
|
|
|
|
|
# ModernBert Model Card
|
|
|
|
|
|
[ModernBert Technical Report](https://arxiv.org/pdf/2412.08905)
|
|
|
|
|
|
## Model Summary
|
|
|
|
|
|
| | |
|
|
|
|-------------------------|-------------------------------------------------------------------------------|
|
|
|
| **Developed by** | Micro |
|
|
|
| **Description** | `phi-4` is a state-of-the-art open model built upon a blend of synthetic datasets, data from filtered public domain websites, and acquired academic books and Q&A datasets. The goal of this approach was to ensure that small capable models were trained with data focused on high quality and advanced reasoning.<br><br>`phi-4` underwent a rigorous enhancement and alignment process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures |
|
|
|
| **Architecture** | 14B parameters, dense decoder-only Transformer model |
|
|
|
| **Inputs** | Text, best suited for prompts in the chat format |
|
|
|
| **Context length** | 16K tokens |
|
|
|
| **GPUs** | 1920 H100-80G |
|
|
|
| **Training time** | 21 days |
|
|
|
| **Training data** | 9.8T tokens |
|
|
|
| **Outputs** | Generated text in response to input |
|
|
|
| **Dates** | October 2024 – November 2024 |
|
|
|
| **Status** | Static model trained on an offline dataset with cutoff dates of June 2024 and earlier for publicly available data |
|
|
|
| **Release date** | March 17, 2025 |
|
|
|
| **License** | MIT |
|
|
|
|
|
|
|
