|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
pipeline_tag: image-text-to-text |
|
|
--- |
|
|
|
|
|
<div align="center"> |
|
|
<picture> |
|
|
<img src="https://github.com/stepfun-ai/Step3/blob/main/figures/stepfun-logo.png?raw=true" width="30%" alt="StepFun: Cost-Effective Multimodal Intelligence"> |
|
|
</picture> |
|
|
</div> |
|
|
|
|
|
<hr> |
|
|
|
|
|
<div align="center" style="line-height:1"> |
|
|
<a href="https://stepfun.com/" target="_blank"><img alt="Chat" src="https://img.shields.io/badge/Chat-StepFun-ff6b6b?color=1783ff&logoColor=white"/></a> |
|
|
<a href="https://stepfun.com/" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-StepFun-white?logo=StepFun&logoColor=white"/></a> |
|
|
</div> |
|
|
|
|
|
<div align="center" style="line-height: 1;"> |
|
|
<a href="https://github.com/stepfun-ai/Step3" target="_blank"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-StepFun-white?logo=github&logoColor=white"/></a> |
|
|
<a href="https://www.modelscope.cn/models/stepfun-ai/step3" target="_blank"><img alt="ModelScope" src="https://img.shields.io/badge/🤖ModelScope-StepFun-ffc107?color=7963eb&logoColor=white"/></a> |
|
|
<a href="https://x.com/StepFun_ai" target="_blank"><img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-StepFun-white?logo=x&logoColor=white"/></a> |
|
|
</div> |
|
|
|
|
|
<div align="center" style="line-height: 1;"> |
|
|
<a href="https://discord.com/invite/XHheP5Fn" target="_blank"><img alt="Discord" src="https://img.shields.io/badge/Discord-StepFun-white?logo=discord&logoColor=white"/></a> |
|
|
<a href="https://huggingface.co/stepfun-ai/step3-fp8/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue?&color=blue"/></a> |
|
|
</div> |
|
|
|
|
|
<div align="center"> |
|
|
<b>📰 <a href="https://stepfun.ai/research/step3">Step3 Model Blog</a></b> | <b>📄 <a href="https://huggingface.co/papers/2507.19427">Paper</a></b> |
|
|
</div> |
|
|
|
|
|
## Introduction |
|
|
|
|
|
Step3 is our cutting-edge multimodal reasoning model—built on a Mixture-of-Experts architecture with 321B total parameters and 38B active. |
|
|
It is designed end-to-end to minimize decoding costs while delivering top-tier performance in vision–language reasoning. |
|
|
Through the co-design of Multi-Matrix Factorization Attention (MFA) and Attention-FFN Disaggregation (AFD), |
|
|
Step3 maintains exceptional efficiency across both flagship and low-end accelerators. |
|
|
|
|
|
### Step3 model card: |
|
|
|
|
|
| Config | Value | |
|
|
|------------------------|---------| |
|
|
| **Number of Layers (Dense layer included)**|61| |
|
|
|**Number of Dense Layers**| 5| |
|
|
| **Hidden Dimension** | 7168 | |
|
|
| **Attention Mechanism** | MFA | |
|
|
| **Low-rank Query Dimension** | 2048 | |
|
|
| **Number of Query Heads** | 64 | |
|
|
| **Head Dimension** | 256 | |
|
|
|**Number of Experts** |48| |
|
|
|**Selected Experts per Token**|3| |
|
|
|**Number of Shared Experts**| 1| |
|
|
| **Max Context Length** | 65536 | |
|
|
| **Tokenizer** | Deepseek V3 | |
|
|
| **Total Parameters (LLM)** | 316B | |
|
|
| **Activated Params per Token** | 38B | |
|
|
| **Total Parameters (VLM)** | 321B | |
|
|
|
|
|
|
|
|
## Evaluation Results |
|
|
<table> |
|
|
<thead> |
|
|
<tr> |
|
|
<th></th> |
|
|
<th>Model</th> |
|
|
<th>Total Params.</th> |
|
|
<th>MMMU</th> |
|
|
<th>MathVision</th> |
|
|
<th>ZeroBench(sub)</th> |
|
|
<th>DYNAMATH</th> |
|
|
<th>SimpleVQA</th> |
|
|
<th>HallusionBench</th> |
|
|
<th>AIME25</th> |
|
|
<th>HMMT25</th> |
|
|
<th>CNMO24</th> |
|
|
<th>GPQA-Diamond</th> |
|
|
<th>LiveCodeBench<br>(24.8-25.5)</th> |
|
|
</tr> |
|
|
</thead> |
|
|
<tbody> |
|
|
<tr> |
|
|
<td rowspan="6">Open-Source VLM</td> |
|
|
<td>Step3</td> |
|
|
<td>321B</td> |
|
|
<td>74.2</td> |
|
|
<td>64.8</td> |
|
|
<td>23.0</td> |
|
|
<td>50.1</td> |
|
|
<td>62.2</td> |
|
|
<td>64.2</td> |
|
|
<td>82.9</td> |
|
|
<td>70.0</td> |
|
|
<td>83.7</td> |
|
|
<td>73.0</td> |
|
|
<td>67.1</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>ERINE4.5 - thinking</td> |
|
|
<td>300B/424B</td> |
|
|
<td>70.0</td> |
|
|
<td>47.6</td> |
|
|
<td>22.5</td> |
|
|
<td>46.9</td> |
|
|
<td>59.8</td> |
|
|
<td>60.0</td> |
|
|
<td>35.1</td> |
|
|
<td>40.5*</td> |
|
|
<td>75.5</td> |
|
|
<td>76.8</td> |
|
|
<td>38.8</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GLM-4.1V-thinking</td> |
|
|
<td>9B</td> |
|
|
<td>68.0</td> |
|
|
<td>49.4</td> |
|
|
<td>22.8</td> |
|
|
<td>41.9</td> |
|
|
<td>48.1</td> |
|
|
<td>60.8</td> |
|
|
<td>13.3</td> |
|
|
<td>6.7</td> |
|
|
<td>25.0</td> |
|
|
<td>47.4</td> |
|
|
<td>24.2</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MiMo-VL</td> |
|
|
<td>7B</td> |
|
|
<td>66.7</td> |
|
|
<td>60.4</td> |
|
|
<td>18.6</td> |
|
|
<td>45.9</td> |
|
|
<td>48.5</td> |
|
|
<td>59.6</td> |
|
|
<td>60.0</td> |
|
|
<td>34.6</td> |
|
|
<td>69.9</td> |
|
|
<td>55.5</td> |
|
|
<td>50.1</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>QvQ-72B-Preview</td> |
|
|
<td>72B</td> |
|
|
<td>70.3</td> |
|
|
<td>35.9</td> |
|
|
<td>15.9</td> |
|
|
<td>30.7</td> |
|
|
<td>40.3</td> |
|
|
<td>50.8</td> |
|
|
<td>22.7</td> |
|
|
<td>49.5</td> |
|
|
<td>47.3</td> |
|
|
<td>10.9</td> |
|
|
<td>24.1</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>LLaMA-Maverick</td> |
|
|
<td>400B</td> |
|
|
<td>73.4</td> |
|
|
<td>47.2</td> |
|
|
<td>22.8</td> |
|
|
<td>47.1</td> |
|
|
<td>45.4</td> |
|
|
<td>57.1</td> |
|
|
<td>19.2</td> |
|
|
<td>8.91</td> |
|
|
<td>41.6</td> |
|
|
<td>69.8</td> |
|
|
<td>33.9</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="4">Open-Source LLM</td> |
|
|
<td>MiniMax-M1-80k</td> |
|
|
<td>456B</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>76.9</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>70.0</td> |
|
|
<td>65.0</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Qwen3-235B-A22B-Thinking</td> |
|
|
<td>235B</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>81.5</td> |
|
|
<td>62.5</td> |
|
|
<td>-</td> |
|
|
<td>71.1</td> |
|
|
<td>65.9</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>DeepSeek R1-0528</td> |
|
|
<td>671B</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>87.5</td> |
|
|
<td>79.4</td> |
|
|
<td>86.9</td> |
|
|
<td>81.0</td> |
|
|
<td>73.3</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Qwen3-235B-A22B-Thinking-2507</td> |
|
|
<td>235B</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>92.3</td> |
|
|
<td>83.9</td> |
|
|
<td>-</td> |
|
|
<td>81.1</td> |
|
|
<td>-</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="6">Proprietary VLM</td> |
|
|
<td>O3</td> |
|
|
<td>-</td> |
|
|
<td>82.9</td> |
|
|
<td>72.8</td> |
|
|
<td>25.2</td> |
|
|
<td>58.1</td> |
|
|
<td>59.8</td> |
|
|
<td>60.1</td> |
|
|
<td>88.9</td> |
|
|
<td>70.1</td> |
|
|
<td>86.7</td> |
|
|
<td>83.3</td> |
|
|
<td>75.8</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Claude4 Sonnet (thinking)</td> |
|
|
<td>-</td> |
|
|
<td>76.9</td> |
|
|
<td>64.6</td> |
|
|
<td>26.1</td> |
|
|
<td>48.1</td> |
|
|
<td>43.7</td> |
|
|
<td>57.0</td> |
|
|
<td>70.5</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>75.4</td> |
|
|
<td>55.9</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Claude4 opus (thinking)</td> |
|
|
<td>-</td> |
|
|
<td>79.8</td> |
|
|
<td>66.1</td> |
|
|
<td>25.2</td> |
|
|
<td>49.3</td> |
|
|
<td>47.2</td> |
|
|
<td>59.9</td> |
|
|
<td>75.5</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>79.6</td> |
|
|
<td>56.6</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Gemini 2.5 Flash (thinking)</td> |
|
|
<td>-</td> |
|
|
<td>73.2</td> |
|
|
<td>57.3</td> |
|
|
<td>20.1</td> |
|
|
<td>57.1</td> |
|
|
<td>61.1</td> |
|
|
<td>65.2</td> |
|
|
<td>72.0</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>82.8</td> |
|
|
<td>61.9</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Gemini 2.5 Pro</td> |
|
|
<td>-</td> |
|
|
<td>81.7</td> |
|
|
<td>73.3</td> |
|
|
<td>30.8</td> |
|
|
<td>56.3</td> |
|
|
<td>66.8</td> |
|
|
<td>66.8</td> |
|
|
<td>88.0</td> |
|
|
<td>-</td> |
|
|
<td>-</td> |
|
|
<td>86.4</td> |
|
|
<td>71.8</td> |
|
|
</tr> |
|
|
<!-- 新增 Grok 4 --> |
|
|
<tr> |
|
|
<td>Grok 4</td> |
|
|
<td>-</td> |
|
|
<td>80.9</td> |
|
|
<td>70.3</td> |
|
|
<td>22.5</td> |
|
|
<td>40.7</td> |
|
|
<td>55.9</td> |
|
|
<td>64.8</td> |
|
|
<td>98.8</td> |
|
|
<td>93.9</td> |
|
|
<td>85.5</td> |
|
|
<td>87.5</td> |
|
|
<td>79.3</td> |
|
|
</tr> |
|
|
</tbody> |
|
|
</table> |
|
|
|
|
|
Note: Parts of the evaluation results are reproduced using the same settings. |
|
|
†: Evaluation results of Gemini 2.5 Flash (thinking) may be lower than real model performance, especially on MathVision, due to insufficient instruction following ability. |
|
|
## Deployment |
|
|
|
|
|
> [!Note] |
|
|
> Step3's API is accessible at https://platform.stepfun.com/, where we offer OpenAI-compatible API for you. |
|
|
|
|
|
### Inference with Hugging Face Transformers |
|
|
|
|
|
We introduce how to use our model at inference stage using transformers library. It is recommended to use python=3.10, torch>=2.1.0, and transformers=4.54.0 as the development environment.We currently only support bf16 inference, and multi-patch for image preprocessing is supported by default. This behavior is aligned with vllm and sglang. |
|
|
|
|
|
|
|
|
```python |
|
|
from transformers import AutoProcessor, AutoModelForCausalLM |
|
|
|
|
|
key_mapping = { |
|
|
"^vision_model": "model.vision_model", |
|
|
r"^model(?!\.(language_model|vision_model))": "model.language_model", |
|
|
"vit_downsampler": "model.vit_downsampler", |
|
|
"vit_downsampler2": "model.vit_downsampler2", |
|
|
"vit_large_projector": "model.vit_large_projector", |
|
|
} |
|
|
|
|
|
model_path = "stepfun-ai/step3" |
|
|
|
|
|
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_path, |
|
|
device_map="auto", torch_dtype="auto",trust_remote_code=True, |
|
|
key_mapping=key_mapping) |
|
|
|
|
|
messages = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"}, |
|
|
{"type": "text", "text": "What's in this picture?"} |
|
|
] |
|
|
}, |
|
|
] |
|
|
|
|
|
inputs = processor.apply_chat_template( |
|
|
messages, add_generation_prompt=True, tokenize=True, |
|
|
return_dict=True, return_tensors="pt" |
|
|
).to(model.device) |
|
|
|
|
|
generate_ids = model.generate(**inputs, max_new_tokens=32768, do_sample=False) |
|
|
decoded = processor.decode(generate_ids[0, inputs["input_ids"].shape[-1] :], skip_special_tokens=True) |
|
|
|
|
|
print(decoded) |
|
|
|
|
|
``` |
|
|
|
|
|
|
|
|
### Inference with vLLM and SGLang |
|
|
|
|
|
|
|
|
Our model checkpoints are stored in bf16 and block-fp8 format, you can find it on [Huggingface](https://huggingface.co/collections/stepfun-ai/step3-688a3d652dbb45d868f9d42d). |
|
|
|
|
|
Currently, it is recommended to run Step3 on the following inference engines: |
|
|
|
|
|
* vLLM |
|
|
* SGLang |
|
|
|
|
|
Deployment and Request examples for vLLM and SGLang can be found in the [Model Deployment Guide](docs/deploy_guidance.md). |
|
|
|
|
|
## Contact Us |
|
|
If you have any questions, please reach out at [contact@stepfun.com](mailto:contact@stepfun.com) . |
|
|
|
|
|
## License |
|
|
Both the code repository and the model weights are released under the [Apache License (Version 2.0)](./LICENSE). |
|
|
|
|
|
## Citation |
|
|
``` |
|
|
@misc{step3system, |
|
|
title={Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding}, |
|
|
author={StepFun Team}, |
|
|
year={2025}, |
|
|
eprint={2507.19427}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.LG}, |
|
|
url={https://arxiv.org/abs/2507.19427}, |
|
|
} |
|
|
|
|
|
@misc{step3blog, |
|
|
title={Step3: Cost-Effective Multimodal Intelligence}, |
|
|
author={StepFun Team}, |
|
|
url={https://stepfun.ai/research/step3}, |
|
|
} |
|
|
``` |
