LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model

This repository contains the LLaVA-OneVision-1.5 models, as presented in the paper LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training.

Project Page: https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5

Code: https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5

LLaVA-OneVision1.5 introduces a novel family of fully open-source Large Multimodal Models (LMMs) that achieves state-of-the-art performance with substantially lower cost through training on native resolution images.

Superior Performance A family of fully open-source large multimodal models demonstrating
- Superior performance across multiple multimodal benchmarks
- outperforming Qwen2.5-VL in most evaluation tasks.
High-Quality Data at Scale Meticulously curated pre-training and SFT data with rigorous filtering and quality control, achieving superior data efficiency with only 64B tokens.
- Concept-balanced, highly diverse, high-quality caption data
- Comprehensive instruction fine-tuning data covering a wide range of tasks
Ultra-Efficient Training Framework Complete end-to-end training framework designed for maximum efficiency:
- $16000 total budget for full model training on A100 GPUs ($0.6 per GPU/Hour)
- Built on MegatronLM with support for MoE, FP8, and long sequence parallelization
- Optimized codebase for cost-effective scaling
Fully Open Framework for community access and reproducibility:
- High-quality pre-training & SFT data
- Complete training framework & code
- Training recipes & configurations
- Comprehensive training logs & metrics

Models

Model	HF Link	Training Log
LLaVA-OV-1.5-4B-Instruct	🤗 HF / 4B-Instruct	📈 Tensorboard
LLaVA-OV-1.5-8B-Instruct	🤗 HF / 8B-Instruct	📈 Tensorboard

Datasets

(a) The vocabulary coverage proportion in the LLaVA-OneVision-1.5 Mid-Training dataset before and after concept balancing. (b) Distribution of data sources within the LLaVA-OneVision-1.5 Mid-Training dataset. (c) Distribution of data sources within the LLaVA-OneVision-1.5 Insturct dataset.

Description	Link	Status
OV-1.5-Mid-Training-85M	🤗HF/85M	Uploading…
OV-1.5-Instruct	🤗HF/Inst	Uploading…

Evaluation Results

All evaluations were conducted using lmms_eval.

	LLaVA-OV-1.5-8B	Qwen2.5 VL 7B
MMMU (Validation)	55.44	51.33
MMMU-Pro (Standard)	37.40	36.30
MMMU-Pro (Vision)	25.15	32.83
MMBench (English; Test)	84.14	83.40
MMBench (Chinese; Test)	81.00	81.61
MME-RealWorld (English)	62.31	57.33
MME-RealWorld (Chinese)	56.11	51.50
AI2D (With Mask)	84.16	82.58
AI2D (Without Mask)	94.11	93.36
CV-Bench	80.82	79.95
VL-RewardBench	45.90	49.65
V*	78.01	76.96
PixmoCount	62.19	63.33
CountBench	88.19	86.35
ChartQA	86.48	84.08
CharXiv (Direct Questions)	74.10	69.80
DocVQA (Test)	95.00	94.93
InfoVQA (Test)	78.42	81.67
WeMath	33.62	33.33
MathVista (Mini)	69.57	68.60
MathVision	25.56	22.37
MMStar	67.72	62.54
SEED-Bench (Image)	77.32	77.53
ScienceQA	94.98	88.75
SEED-Bench 2-Plus	69.21	70.93
OCRBench	82.90	84.20
RealWorldQA	68.10	68.50

Using 🤗 Transformers to Chat

Here we show a code snippet to show you how to use the chat model with transformers and qwen_vl_utils:

from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
from qwen_vl_utils import process_vision_info
model_path = "lmms-lab/LLaVA-OneVision-1.5-8B-Instruct"

# default: Load the model on the available device(s)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
)

# default processer
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

Citation

If you find LLaVA-OneVision-1.5 useful in your research, please consider to cite the following related papers:

@misc{an2025llavaonevision15fullyopenframework,
      title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, 
      author={Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
      year={2025},
      eprint={2509.23661},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.23661}, 
}

@inproceedings{xie2025region,
  title={Region-based Cluster Discrimination for Visual Representation Learning},
  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
  booktitle={ICCV},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research}
  year={2024}
}