acceptodds
Withdrawn from ICLR 2027

OneResearchBench: Benchmarking Research Agents for Research Synthesis from Real-World Source Materials

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Progress toward general-purpose AI agents requires capabilities beyond isolated question answering. Agents must operate reliably across diverse domains, handle complex user-provided materials, and complete long-horizon knowledge tasks that involve gathering evidence, integrating information, and producing trustworthy outputs. Existing research-agent benchmarks, however, largely start from research questions or open information needs, and give limited attention to research that begins from real-world source materials. We define source-grounded research synthesis, a task in which an agent must first understand provided materials, then conduct external research, integrate the resulting evidence, and produce a grounded long-form report. To evaluate this capability, we introduce OneResearchBench, containing 101 human-constructed tasks across six domains and 3,671 expert-written, task-specific rubric rules. The rubrics evaluate source understanding, external research, evidence grounding, technical accuracy, citation traceability, and structure quality. Our criterion-level evaluator achieves a Pearson correlation of 0.909 with expert scores on the human calibration subset. We evaluate eight model–harness configurations. Under direct report generation, no system scores above 44.75/100, with external research and citation traceability emerging as the largest capability gaps. A structured skill-driven research workflow improves Claude Opus 4.8 with Claude Code from 44.75 to 53.07 (+8.32), and GPT-5.5 with Codex from 43.03 to 47.90 (+4.87). These results show that source-grounded research synthesis remains challenging for current research agents. OneResearchBench provides a benchmark for measuring progress on this task.

Primary area:
datasets and benchmarks
Keywords:
Research Agents, Agent Skills, Deep Research

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vHUwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OneResearchBench to evaluate a research agent's ability to conduct external research based on user-provided source material, integrate evidence, and generate long-form research reports. The benchmark consists of 101 tasks in six areas and 3,671 rubric rules written by experts covering six assessment dimensions. The authors evaluated eight model–harness configurations and verified the reliability of the LLM judge through human expert ratings. The results show that there are still significant shortcomings in current research agents, especially in terms of external research and citation traceability.

Critical strengths

The paper focuses on research synthesis tasks based on real source materials, has practical application value, and focuses on examining external information retrieval and evidence integration capabilities. Scoring rubrics written by experts for each task evaluate the research agent from six dimensions, helping to identify shortcomings in its capabilities.

Critical weaknesses

W1. ExpertLongBench has used fine-grained scoring rules and the CLEAR framework designed by experts to evaluate the factual accuracy and source consistency of long text generation, which highly overlaps with the Source Understanding and Evidence Grounding dimensions of this article. While OneResearchBench clearly emphasizes external research and evidence integration, the authors do not adequately address what capability gaps these additions reveal that existing benchmarks fail to uncover, or what substantive new findings they lead to. Therefore, the need to propose this benchmark alone remains unclear.

W2. Although the Pearson correlation coefficient between LLM judges and human expert ratings is high, the correlation at the report level does not prove the accuracy of the item-by-item ratings. In particular, dimensions such as external research quality and citation traceability require explicit verification of evidence. The paper lacks independent validation and detailed error analysis for these key dimensions.

Questions for authors

Q1. What unique capability shortcomings or agent behaviors does OneResearchBench reveal compared to existing benchmarks such as ExpertLongBench? Can the authors demonstrate through controlled experiments the necessity of the new assessment dimensions and what substantive new findings they bring about?

Q2. Can the author provide an analysis of the consistency between LLM judges and manual ratings in each evaluation dimension, and specifically evaluate the judgment accuracy of LLM judges in terms of external evidence verification and citation traceability?

Reviewer_xNFjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for writing research reports from user-provided materials and outside evidence. It has 101 tasks across six fields and 3,671 scoring rules written by experts. An LLM judge scores the reports against the rules. The authors compare model scores with expert scores on 24 tasks and 120 reports. They also test a skill that divides the work into reading, search, writing, and review. This skill raises the overall score for two model setups, although some dimension scores fall.

Critical strengths

I think the task is useful. In real research work, users often start with files rather than a single question. I vote for the requirement to separate facts from those files, outside evidence, and the agent's own claims. The tasks cover a useful range of fields.

Critical weaknesses
  • I am not convinced by the pooled correlation alone. I recalculated Table 12 from the rounded scores. On the same 24 tasks, the judge ranks Claude, GPT, then Qwen. The experts rank GPT, Qwen, then Claude. The judge scores Claude 11.37 points above experts on average, compared with 7.67 points for GPT. I reproduce the correlation of 0.909, but the scoring gap depends on the system. I find this worrying for close ranks. This does not prove that the full 101-task ranking is wrong.
  • I think the skill comparison is weak. The direct prompt asks for outside context when useful. The skill gives clear instructions for search, checks, and repeated revision. Both have the same goal, but the guidance differs. I do not see a strong direct prompt with matched call and revision budgets. I cannot separate the benefit of the workflow design from stronger instructions or different research effort.
  • I find the technical-score drops serious. With the skill, Technical Accuracy falls from 56.7 to 34.6 for Claude and from 49.6 to 23.9 for GPT. Source Understanding falls too. The authors admit this in the appendix. I think this trade-off should be clearer in the main conclusions. TA is a net rule score that includes penalties and missing details. It is not the share of correct facts, so these drops do not tell us the actual fact error rate.
  • I do not find the novelty clear enough. DR³-Eval already combines user files and outside evidence for report writing. It uses a fixed research corpus, while this paper uses the open web. That is a difference, but the paper needs to explain the added contribution better. I also did not find controls that hide or replace the input files. Without these checks, I am unsure how much the tasks require those files rather than general knowledge of the topic.
Questions for authors
  • Can you explain the rank change in Table 12? Please show scoring bias by system and paired uncertainty across tasks, rather than pooled correlation alone.
  • Can you compare the skill with a stronger direct prompt under matched research, call, and revision budgets? Which stages account for the gain?
  • What causes the lower TA and Source Understanding scores? Do you have a separate human check of factual errors and faithful use of the source materials?
  • What is the main task difference from DR³-Eval beyond live web access? Have you tested performance after hiding or replacing the supplied materials?
Reviewer_NpgSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

OneResearchBench evaluates source-grounded research synthesis through 101 human constructed tasks and 3,671 expert written rubric criteria. The task covers law, arts, stem, econ, medicine and life science and humanity. The paper also compares eight model harness configurations and examines whether a structured skill driven workflow improves report quality: the answer is yes, even for most frontier agents of Claude Code and Codex.

Critical strengths

S1: Timely and important topics: As LLM agents become increasingly capable, robust evaluation requires realistic, long horizon benchmarks for research and scientific tasks. This work has the potential to contribute to that effort and provide value to the broader community.

S2: The benchmark includes expert written rubric criteria and an automated data curation pipeline, providing a useful foundation for systematic evaluation. Open source real expert rubric provides add-on value to the community.

S3: The paper evaluates several major agent harnesses and provides extensive failure analysis and cross domain comparisons.

Critical weaknesses

W1: Limited evidence of dataset quality. In my view, data quality is the central "successful metric" for a dataset and benchmark paper. However, the paper provides insufficient information for readers to assess it. Neither the main text nor the appendix presents concrete task trajectories or sufficient evidence supporting the quality of the tasks. There also appears to be no supplementary dataset or anonymous link through which reviewers can inspect the data. Consequently, even after carefully reading the paper, I find it difficult to judge whether the tasks capture meaningful, high quality, long horizon research tasks.

W2: Insufficient detail on expert annotation and quality control. The benchmark spans six broad research domains and relies on expert written rubrics, which is potentially valuable, especially when open source. However, the paper provides limited information about the experts’ qualifications, the instructions they received, and the procedures used to ensure annotation quality and consistency. These details are essential for assessing the credibility of the rubrics.

W3: Misalignment between baseline instructions and evaluation requirements. External research is presented as optional in the baseline instructions, but external research and citation traceability account for the largest observed capability gaps. This raises concerns about whether the evaluation reflects differences in agent capabilities or differences in how clearly the instructions communicate the evaluated requirements. The paper does not sufficiently explain how it establishes a fair and solid evaluation playground across harnesses.

Questions for authors

Q1 (regarding W1 and W2): Could the authors provide concrete task examples, representative research trajectories, and more detailed evidence of dataset quality? Provide some discussion about eyeballing some data. Could authors also clarify the experts’ backgrounds, annotation instructions, and rubric quality-control procedures? These details are particularly important for a benchmark intended to evaluate long-horizon research and scientific tasks.

Q2 (regarding W3): What principles and design choices guide the construction of a fair and solid environment across agent harnesses? In particular, how do the authors ensure that baseline instructions are aligned with the evaluation criteria and that observed performance differences reflect agent capabilities rather than differences in task guidance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.