- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SuperResearch Bench introduces 300 research tasks with source-linked evidence graphs, reference reports, and evaluation questions. It evaluates 14 systems on coverage, logical consistency, report utility, and objectivity, aiming to assess demanding research tasks.
Critical strengths
- The benchmark provides substantial reference material across ten domains, with source-linked graphs that make the supporting evidence inspectable.
- The documented reference corrections, coverage breakdowns, and qualitative cases provide useful starting points for understanding system failures.
Critical weaknesses
-
Task difficulty is insufficiently justified. Several questions are underspecified, and the reference pipeline’s retrieval volume does not demonstrate that answering them requires unusually extensive research.
-
Reference reliability is insufficiently quantified. Although expert auditing is described, its coverage and residual error rates are not reported for reference facts, inference links, and stance labels.
-
Evaluator accuracy lacks direct validation. Perturbation sensitivity and consistency across judges do not establish agreement with independent experts on report quality.
-
Reference-based scoring may penalize valid alternatives. The analysis does not distinguish missing essential evidence from reasonable differences in sources, scope, or conclusions.
-
The benefit of graph-based evaluation remains unclear. A comparison with flat rubric-based scoring using the same reference evidence is needed to establish its added value.
Questions for authors
-
More models should be tested.
-
How are valid answers using evidence or reasoning absent from the reference graph evaluated?
-
Can the authors report paired confidence intervals for the leading systems and clarify comparability across evaluation dates and search budgets?