LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

Excited to share our work on deep research!
In this work, we argue that four task design principles are essential for fair comparison in deep research: (1) user-centric, (2) dynamic, (3) unambiguous, and (4) multi-faceted & search-intensive, and LiveResearchBench is guided entirely by these. It includes 100 expert-curated questions with detailed checklists, built through 1,500+ hours of human effort and an 11-step curation and validation pipeline to ensure the benchmark truly satisfies those criteria.
Evaluating open-ended, long-form reports is equally challenging. Simply assigning scores or preferences using a single LLM-as-a-judge leads to unstable, unreliable model assessments. To address this, we propose DeepEval, a framework with six metrics that make long-form evaluation robust—so models can’t “hack” their outputs by generating shorter or safer reports. Using LLM-ensemble-as-judge, each metric follows a tailored protocol: checklist-based for presentation & coverage, pointwise (additive) for consistency & citation association, pairwise comparison for depth of analysis, and rubric-tree for citation accuracy. This setup yields stable assessment and high agreement with human experts!
With LiveResearchBench and DeepEval, we benchmark 17 open and proprietary agentic systems, uncovering current strengths, recurring failure modes, and many interesting insights, including that most systems today act more like deep searchers rather than deep researchers.
One open challenge we find especially intriguing: for multi agent systems, scaling up number of subagents and tool calls can easily reach thousands (>3,000) webpages. How can we compress this information without losing critical evidence? When content is partially redundant, how do we merge overlaps without discarding unique signals? And when everything is relevant and important but still exceeds context limits, how do we retain the essentials?
More details in the paper--you won’t regret it!