- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper propose a knowledge-centric benchmark for deep research agents. The authors design an automated data annotation pipeline from wikidata and construct the benchmark with 144 cases across 24 domains and 44,102 rubrics. Besides, they provide an interesting evaluation framework by measuring the breadth across major research directions and depth along related knowledge chains. The results reveals different behaviors of different DR agents and the evaluation is closed to the human annotators.
Critical strengths
- The automated data pipeline is effective in generating large-scale DR instructions and answers using authoritative information sources from Wikidata, with minimal human intervention.
- The evaluation goals reflect the objectives of deep research to some extent and provide a new way to measure research performance by calculating its breadth and depth.
Critical weaknesses
- The main concern is about the evaluation protocol. The depth metric is intended to reflect continuous reasoning in the report. However, according to the evaluation prompt, the evaluation relies on independently judging each rubric. Such point-level evaluation cannot guarantee that the corresponding reasoning is actually coherent in the report. For example, both a leaf-node rubric and its parent-node rubric may be satisfied by content appearing in different sections of the report, without forming an actual reasoning chain or demonstrating meaningful depth. Nevertheless, the metric may still consider them as continuous coverage. This raises significant concerns about the validity of the depth metric.
- Although the paper provides case studies, including the instructions and scores of some DR agents, we do not see any actual examples of the rubrics or the generated DR reports.
- Although the paper contains a large number of rubrics for each instruction, it is unclear whether they can cover all possible valid reasoning paths. For example, a report may reach valid conclusions through alternative reasoning paths or different evidence that are not included in the rubric tree. Therefore, the construction of the rubric tree may overlook some valid reasoning possibilities.
Questions for authors
-
I wonder whether the issue mentioned in Weakness 1 actually occurs in the benchmark cases and whether it affects the evaluation scores.
-
Could the authors provide more examples of the actual reports generated by DR agents and their corresponding rubric satisfaction results? The current aggregate metrics alone do not provide an intuitive understanding of the quality of the generated reports.
6 comments
Rebuttal [2/3]
[Weakness 2 / Question 2]: Could the authors provide more examples of the actual reports generated by DR agents and their corresponding rubric satisfaction results?
Thank you for your suggestion. We have added a concrete report–rubric example in Appendix E.4 (Example Showcase). We briefly describe the example and its rubric satisfaction results below.
The root entity is Roguelike, a video game genre typically featuring procedurally generated dungeons and permadeath. The query asks for an analysis of the genre's origins, covering early games, technical constraints, tabletop influences, and distribution. We examine o3 Deep Research's report.
The selected relations are shown below, with entities as nodes and rubrics as edges (✓: satisfied; ✗: unmet). R10–R11 provide the path context for R12.
Roguelike
├─[R1 ✓]→ Rogue
│ ├─[R2 ✓]→ Permadeath
│ │ ├─[R3 ✓]→ Save scumming
│ │ └─[R4 ✓]→ Tabletop games
│ └─[R5 ✓]→ Hack
│ └─[R6 ✗]→ Porting
└─[R10 ✓]→ Beneath Apple Manor
└─[R11 ✓]→ Procedural generation
└─[R12 ✗]→ Random seed
Shared prefix omitted: “The report explicitly states or clearly paraphrases that”.
| ID / Satisfied | Rubric requirement |
|---|---|
| R1 ✓ | The 1980 game Rogue is considered the forerunner and namesake of the roguelike genre. |
| R2 ✓ | Rogue implements permadeath, meaning a character that loses all health is dead. |
| R3 ✓ | Players may circumvent permadeath by backing up and restoring savefiles, a tactic called 'save scumming' that is considered cheating. |
| R5 ✓ | Rogue inspired the development of Hack. |
| R6 ✗ | Hack was ported to other platforms, with some later ports using graphical tiles instead of ASCII characters. |
| R12 ✗ | Procedural generation used predefined seed values (map seeds) to generate very large game worlds. |
Depth and continuity along knowledge chains. Along R1→R2→R3, the report progresses from Rogue’s role as the genre’s forerunner and namesake to permadeath, then to ways of circumventing and enforcing this mechanic. The following two consecutive sentences from the report support rubrics R2 and R3, respectively:
In Rogue, death was easy to come by and meant starting over from scratch, which forced players to focus on mastering the game’s systems rather than any single lucky run. (Notably, when some crafty students found they could copy save files to cheat death, Rogue’s authors changed the code to delete the save upon loading, enforcing permadeath.)
The next sentence adds the D&D connection (R4). Together, R2–R4 show progressively more specific coverage within one paragraph, with the intermediate relations covered.
Informative omissions. For R6, the report discusses graphical tiles for Beneath Apple Manor, not Hack, leaving the platform and display relation uncovered. For R12, it describes randomized dungeons without explaining the seed mechanism, a further technical extension. Despite being unmet, these rubrics remain valuable for the task, helping distinguish the specificity and completeness of reports’ knowledge coverage.
For R1→R5, evidence spans different sections while maintaining a clear historical connection: R1 is satisfied in the section that describes how Rogue was developed and widely distributed, explaining how it became the genre’s defining model. R5 is satisfied in the subsequent section that discusses early roguelike variants and their developers, describing how Hack’s creator attempted to recreate Rogue from memory. The discussion progresses from establishing Rogue’s foundational role to showing how its design influenced a specific successor, preserving the logical connection across the section boundary.
We sincerely appreciate your suggestion, which helped us further refine our paper.
Rebuttal [3/3]
[Weakness 3]: For example, a report may reach valid conclusions through alternative reasoning paths or different evidence that are not included in the rubric tree.
Thank you for your question. The benchmark's reference trees and rubrics are constructed from Wikipedia, and agents in our local research framework retrieve evidence over a fixed Wikipedia snapshot. Queries and rubrics originate from the same selected subgraph. Together, these components have already established a closed-loop evaluation within the local research framework, connecting construction, retrieval, and scoring through a common knowledge source.
Regarding the alternative reasoning paths or evidence you mention, it is important to first recognize that a further challenge concerns the completeness of the rubric tree’s coverage. For open-ended research, a finite reference cannot reasonably be expected to enumerate every valid research direction or cover all potentially relevant sources across the continually expanding open web. For example, AutoResearchBench [1] acknowledges that exhaustive answer sets remain difficult even with rigorous verification, while DeepResearch Bench [2] notes the difficulty of establishing a definitive standard answer for complex research queries. These works likewise ground evaluation in reference answers constructed from a finite body of source material, or in model-generated references, while acknowledging the difficulty of establishing exhaustive coverage.
More importantly, however, our approach to reference construction and structured evaluation is general in design and can be extended to other knowledge sources. The alternative reasoning paths or evidence you mention may draw on academic papers, industry reports, government records, or other sources. If we had sufficient access to these materials, along with the resources and permissions needed to process and reuse them, we could incorporate the relevant entities and relationships as new tree nodes and branches, yielding more comprehensive reference trees and rubrics.
For the current benchmark, we chose Wikipedia as one of the most readily accessible, openly licensed encyclopedic resources, offering public data dumps and clear reuse terms that facilitate large-scale benchmark construction and reproducible local retrieval. Wiki Live Challenge [3] and FreshWiki [4] also use Wikipedia articles as references, while TRQA’s Wiki subset [5] pairs structured knowledge with a Wikipedia corpus. These precedents also support our choice of Wikipedia as a reasonable foundation for benchmark construction and evaluation.
Ref:
[1] Xiong et al. AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery.
[2] Du et al. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents.
[3] Wang et al. Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles.
[4] Shao et al. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models.
[5] Rafiee et al. Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents.
Thank you again for the thorough review. We hope our responses address your concerns. If you have any further questions, we would be happy to answer them.
Sincerely,
DRBench Pro Authors
Rebuttal [1/3]
Dear Reviewer eean,
Thank you for the effort and time you dedicated to reviewing our paper, and for your recognition of the contribution of our benchmark and its evaluation approach. We have greatly benefited from your valuable feedback. Below are our responses to the questions you pointed out:
Thank you for your question. Independent rubric judgments are the inputs to our depth evaluation, while the aggregation preserves their structural relationships. To address your concern that “both a leaf-node rubric and its parent-node rubric may be satisfied by content appearing in different sections of the report,” we also examined the locations of the supporting evidence for consecutive satisfied rubrics to assess their textual proximity. Our analysis found this pattern relatively uncommon in the sampled cases.
We analyzed a selected set of 500 sampled reports. For groups of consecutive rubrics originally judged satisfied, we used Qwen3.8 Flash to locate supporting evidence sentences and checked whether they appeared in the same main body section. The rates below are averaged equally across reports.
In this subset, evidence for both chain lengths is predominantly located within the same main body section. We further inspected examples where the supporting evidence appeared in different sections and found that, in some cases, knowledge points in a coherent logical chain do not necessarily have to appear consecutively in the text. Reports may devote separate sections to different explanatory steps, analytical dimensions, or historical stages, or introduce a phenomenon in one section before tracing its origins in another. In such cases, mechanisms, supporting evidence, sources, or applicability conditions can connect explanations across sections and sustain meaningful progression.
Appendix E.4 provides a concrete example of consecutive parent–child rubrics satisfied across different report sections. We briefly describe this example here.
The research query is centered on Roguelike, a video game genre. The selected chain is shown below, with entities as nodes and rubrics as edges:
For R1→R5, evidence spans different sections while maintaining a clear historical reasoning coherence: R1 is satisfied in the section that describes how Rogue was developed and widely distributed, explaining how it became the genre’s defining model. R5 is satisfied in the subsequent section that discusses early roguelike variants and their developers, describing how Hack’s creator attempted to recreate Rogue from memory. The discussion progresses from establishing Rogue’s foundational role to showing how its design influenced a specific successor, preserving the continuous reasoning across the section boundary and thus demonstrating meaningful depth.
Further details are also provided in our separate response to [Weakness 2 / Question 2].