- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OneResearchBench to evaluate a research agent's ability to conduct external research based on user-provided source material, integrate evidence, and generate long-form research reports. The benchmark consists of 101 tasks in six areas and 3,671 rubric rules written by experts covering six assessment dimensions. The authors evaluated eight model–harness configurations and verified the reliability of the LLM judge through human expert ratings. The results show that there are still significant shortcomings in current research agents, especially in terms of external research and citation traceability.
Critical strengths
The paper focuses on research synthesis tasks based on real source materials, has practical application value, and focuses on examining external information retrieval and evidence integration capabilities. Scoring rubrics written by experts for each task evaluate the research agent from six dimensions, helping to identify shortcomings in its capabilities.
Critical weaknesses
W1. ExpertLongBench has used fine-grained scoring rules and the CLEAR framework designed by experts to evaluate the factual accuracy and source consistency of long text generation, which highly overlaps with the Source Understanding and Evidence Grounding dimensions of this article. While OneResearchBench clearly emphasizes external research and evidence integration, the authors do not adequately address what capability gaps these additions reveal that existing benchmarks fail to uncover, or what substantive new findings they lead to. Therefore, the need to propose this benchmark alone remains unclear.
W2. Although the Pearson correlation coefficient between LLM judges and human expert ratings is high, the correlation at the report level does not prove the accuracy of the item-by-item ratings. In particular, dimensions such as external research quality and citation traceability require explicit verification of evidence. The paper lacks independent validation and detailed error analysis for these key dimensions.
Questions for authors
Q1. What unique capability shortcomings or agent behaviors does OneResearchBench reveal compared to existing benchmarks such as ExpertLongBench? Can the authors demonstrate through controlled experiments the necessity of the new assessment dimensions and what substantive new findings they bring about?
Q2. Can the author provide an analysis of the consistency between LLM judges and manual ratings in each evaluation dimension, and specifically evaluate the judgment accuracy of LLM judges in terms of external evidence verification and citation traceability?