jeepliu/DocReward-3B

[ACL 2026] DocReward: A Document Reward Model for Structuring and Stylizing

Introduction

Recent agentic workflows have automated professional document generation but focus narrowly on textual quality, overlooking structural and stylistic professionalism that is equally critical for readability. This gap stems mainly from a lack of effective reward models capable of guiding agents toward producing documents with high structural and stylistic professionalism. We introduce DocReward, a Document Reward Model that evaluates documents based on their structure and style. To achieve this, we propose a textual-quality-agnostic framework that ensures assessments are not confounded by content quality, and construct DocPair, a dataset of 117K paired documents, covering 32 domains and 267 types. DocReward is trained using the Bradley-Terry loss. On a manually annotated benchmark, DocReward outperforms GPT-5 by 14.6 percentage points in accuracy. Reinforcement learning experiments further show that DocReward effectively guides agents toward generating documents of greater structural and stylistic quality.

Code Repository

For more detailed inference examples and code, please refer to our GitHub repository:

Model Download

We provide two versions of DocReward models on Hugging Face:

• DocReward-3B: https://huggingface.co/jeepliu/DocReward-3B
• DocReward-7B: https://huggingface.co/jeepliu/DocReward-7B

Citation

If you find this work helpful, please cite out paper:

@misc{liu2025docrewarddocumentrewardmodel,
      title={DocReward: A Document Reward Model for Structuring and Stylizing}, 
      author={Junpeng Liu and Yuzhong Zhao and Bowen Cao and Jiayu Ding and Yilin Jia and Tengchao Lv and Yupan Huang and Shaohan Huang and Nan Yang and Li Dong and Lei Cui and Tao Ge and Xun Wang and Huitian Jiao and Sun Mao and FNU Kartik and Si-Qing Chen and Wai Lam and Furu Wei},
      year={2025},
      eprint={2510.11391},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.11391}, 
}