acceptodds
Withdrawn from ICLR 2027

SuperResearch Bench: Stress-Testing Long-Horizon Research with Evidence Graphs and Rubric-Guided Audits

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Large Language Models (LLMs) have demonstrated proficiency in Deep Research or Wide Search, their capacity to solve highly complex questions requiring long-horizon planning, massive evidence gathering, and synthesis across heterogeneous sources remains largely unexplored. We introduce SuperResearch Bench, a benchmark for Super Research: complex autonomous research tasks that integrate (i) structured decomposition, (ii) super wide retrieval for diverse perspectives, and (iii) super deep investigation through iterative queries. To evaluate this capability, we curated 300 expert-written questions across diverse domains, each requiring up to 100+ retrieval steps and 1,000+ web pages to reconcile conflicting evidence. SuperResearch Bench provides source-linked evidence graphs, reference reports with fine-grained citations, and intermediate artifacts to support traceable reasoning. The benchmark artifacts undergo automated validation and expert review. Our graph-anchored, rubric-guided auditing protocol evaluates generated reports along four primary dimensions: Coverage, Logical Consistency, Report Utility, and Objectivity. Citation Health is reported separately. While such questions may be infrequent in standard applications, SuperResearch Bench serves as a ceiling evaluation and stress test for long-horizon research. A comparison of fourteen systems and protocols reveals substantial variation in evidence recovery, reasoning consistency, and report usefulness, enabling fine-grained diagnosis of current research systems.

Primary area:
datasets and benchmarks
Keywords:
research benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KXjSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SuperResearch Bench introduces 300 research tasks with source-linked evidence graphs, reference reports, and evaluation questions. It evaluates 14 systems on coverage, logical consistency, report utility, and objectivity, aiming to assess demanding research tasks.

Critical strengths
  • The benchmark provides substantial reference material across ten domains, with source-linked graphs that make the supporting evidence inspectable.
  • The documented reference corrections, coverage breakdowns, and qualitative cases provide useful starting points for understanding system failures.
Critical weaknesses
  • Task difficulty is insufficiently justified. Several questions are underspecified, and the reference pipeline’s retrieval volume does not demonstrate that answering them requires unusually extensive research.

  • Reference reliability is insufficiently quantified. Although expert auditing is described, its coverage and residual error rates are not reported for reference facts, inference links, and stance labels.

  • Evaluator accuracy lacks direct validation. Perturbation sensitivity and consistency across judges do not establish agreement with independent experts on report quality.

  • Reference-based scoring may penalize valid alternatives. The analysis does not distinguish missing essential evidence from reasonable differences in sources, scope, or conclusions.

  • The benefit of graph-based evaluation remains unclear. A comparison with flat rubric-based scoring using the same reference evidence is needed to establish its added value.

Questions for authors
  • More models should be tested.

  • How are valid answers using evidence or reasoning absent from the reference graph evaluated?

  • Can the authors report paired confidence intervals for the leading systems and clarify comparability across evaluation dates and search budgets?

Reviewer_ST1EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper contributes SuperResearch Bench, a new benchmark that evals models on their ability to iteratively search across both depth and breadth of various sources and assemble a coherent, well-informed research report. This is claimed to be the first benchmark that measures search prowess across both depth and breadth, at the scale of the specific tasks here (hundreds of retrieval steps and thousands of web pages). Contributions and validations as I interpreted them:

  • At task creation, a source-linked evidence graph is created from the decomposition of a task to evaluate generated reports along the axes of coverage, logical consistency, utility, and objectivity.
  • Another model uses the generated reports to answer questions (Q&A), which is a proxy for the effectiveness of the report.
Critical strengths

Two core strengths of the paper: each of the tasks ship with actual evidence graphs, reference (golden) reports, and the task-specific questions. Also, the experimentation with experts testing the evaluator was interesting to me.

The source-linked evidence graphs make it easier to trust the quality of the individual tasks in the benchmark and make them auditable. The experimentation with experts testing the evaluator’s reaction to tweaks in the report means that human eyes fell upon the reviewer and it survived stress-testing. This makes it easier to trust as a measure of quality of performance.

Critical weaknesses

Most tested models are now outdated! The most recent / SOTA models tested are not frequently used in production much anymore. Would want to see more recent model scores and see how close the benchmark is to saturation.

Questions for authors

How is this materially different than DeepWideSearch, in more detail?

Reviewer_FxL9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SuperResearchBench, a super-wide and deep benchmark. According to this paper, evaluating super research is challenging because LLM-as-a-judge has limitations, and human evaluation is expensive. SuperResearchBench contains three advantages: 1) Structured Claim Verification, 2) Interpretable Error Tracing, and 3) Reproducible Scoring. The data quality of this paper is concerning, according to the appendix.

Critical strengths

This paper is working towards complicated, super deep, and super wide deep research. This is an interesting direction.

Critical weaknesses
  1. Professional Precision and Expert Refinement Line 2325, ID 222, Society & Humanities. How does the Riemannian metric of universality measures affect the conformal transformation of local justice traditions? Riemannian metric, universality measures, conformal transformation, and local justice traditions are not related to each other. Similar to this paper (https://physics.nyu.edu/faculty/sokal/transgress_v2/transgress_v2_singlefile.html). ID 175: Society & Humanities. In restorative justice games, how do eigenvectors of punishment matrices determine the stability of social harmony gradients? ID 227: Society & Humanities: How does the Jordan form of social stratification matrices delimit the ergodicity boundary of mobility functions in counter-globalization flow fields? How can we understand these deep research questions?

  2. This paper claims that SuperResearch Bench requires 100+retrieval steps and 1,000+ webpages. According to Figure 6, the number of Explicit HTTP URLs is less than 60. Perplexity Pro Search only uses 14 seconds and writes 5.8k characters, while the reported average length is 80,389. Perplexity Pro achieves the best Objectivity score.

  3. Tongyi Deep Research does not have web search at all, but outperforms OpenAI o3 DeepResearch.

Questions for authors
  1. Data quality vs. claimed professional precision and expert refinement Line 2325, ID 222, Society & Humanities. How does the Riemannian metric of universality measures affect the conformal transformation of local justice traditions? Riemannian metric, universality measures, conformal transformation, and local justice traditions are not related to each other. Similar to this paper (https://physics.nyu.edu/faculty/sokal/transgress_v2/transgress_v2_singlefile.html). ID 175: Society & Humanities. In restorative justice games, how do eigenvectors of punishment matrices determine the stability of social harmony gradients? ID 227: Society & Humanities: How does the Jordan form of social stratification matrices delimit the ergodicity boundary of mobility functions in counter-globalization flow fields? How can we understand these deep research questions? Question: Please explain what these questions mean. Please note that tasks with such issues are not limited to the examples provided. As a deep research task, the tasks with problems do not have corresponding evidence available on the internet.

  2. This paper claims that SuperResearch Bench requires 100+retrieval steps and 1,000+ webpages. According to Figure 6, the number of Explicit HTTP URLs is less than 60. Perplexity Pro Search only uses 14 seconds and writes 5.8k characters, while the reported average length is 80,389. Perplexity Pro achieves the best Objectivity score. Question: Why can such short deep research lead to good performance? Why is the provided report length shorter than claimed?

  3. Tongyi Deep Research does not have web search at all, but outperforms OpenAI o3 DeepResearch. Question: Does the evaluation system have design issues? Why does Tongyi Deep Research with zero HTTP URLs can outperform OpenAI o3 DeepResearch with 3.5 URLs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.