acceptodds
Under review as a conference paper at ICLR 2027

DRBench Pro: A Knowledge-Centric Benchmark for Professional Deep Research Agents

Abstract

Deep research agents gather information from multiple sources and synthesize it into reports on complex, open-ended questions. Existing benchmarks assess report quality along several dimensions, but lack a dedicated knowledge-centric evaluation of which research directions a report covers and how deeply it explores them. We introduce DRBench Pro, a benchmark that makes structured knowledge coverage the primary evaluation target. Our automated pipeline builds local knowledge trees from Wikipedia and uses sampled subgraphs to generate research queries and evidence-grounded rubrics. Each rubric corresponds to a relation in the tree, preserving the structure of the knowledge being evaluated. Agents receive only the research query. This structure allows us to measure breadth across major research directions and depth along related knowledge chains, with their geometric mean giving overall coverage. The benchmark includes 144 cases across 24 domains and 44,102 rubrics. Evaluating 16 deep research agents reveals different strengths in breadth and depth, with the best-performing agent reaching 57.04% overall coverage. Rubric judgments agree closely with human annotations, and agent rankings remain stable across the tested metric configurations.

Primary area:
datasets and benchmarks
Keywords:
deep research agents, large language model evaluation, long-form report evaluation, structured knowledge coverage, graph-structured benchmarks, breadth and depth metrics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eeanOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper propose a knowledge-centric benchmark for deep research agents. The authors design an automated data annotation pipeline from wikidata and construct the benchmark with 144 cases across 24 domains and 44,102 rubrics. Besides, they provide an interesting evaluation framework by measuring the breadth across major research directions and depth along related knowledge chains. The results reveals different behaviors of different DR agents and the evaluation is closed to the human annotators.

Critical strengths
  1. The automated data pipeline is effective in generating large-scale DR instructions and answers using authoritative information sources from Wikidata, with minimal human intervention.
  2. The evaluation goals reflect the objectives of deep research to some extent and provide a new way to measure research performance by calculating its breadth and depth.
Critical weaknesses
  1. The main concern is about the evaluation protocol. The depth metric is intended to reflect continuous reasoning in the report. However, according to the evaluation prompt, the evaluation relies on independently judging each rubric. Such point-level evaluation cannot guarantee that the corresponding reasoning is actually coherent in the report. For example, both a leaf-node rubric and its parent-node rubric may be satisfied by content appearing in different sections of the report, without forming an actual reasoning chain or demonstrating meaningful depth. Nevertheless, the metric may still consider them as continuous coverage. This raises significant concerns about the validity of the depth metric.
  2. Although the paper provides case studies, including the instructions and scores of some DR agents, we do not see any actual examples of the rubrics or the generated DR reports.
  3. Although the paper contains a large number of rubrics for each instruction, it is unclear whether they can cover all possible valid reasoning paths. For example, a report may reach valid conclusions through alternative reasoning paths or different evidence that are not included in the rubric tree. Therefore, the construction of the rubric tree may overlook some valid reasoning possibilities.
Questions for authors
  1. I wonder whether the issue mentioned in Weakness 1 actually occurs in the benchmark cases and whether it affects the evaluation scores.

  2. Could the authors provide more examples of the actual reports generated by DR agents and their corresponding rubric satisfaction results? The current aggregate metrics alone do not provide an intuitive understanding of the quality of the generated reports.

6 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Dear Reviewer eean,

Thank you for the effort and time you dedicated to reviewing our paper, and for your recognition of the contribution of our benchmark and its evaluation approach. We have greatly benefited from your valuable feedback. Below are our responses to the questions you pointed out:


[Weakness 1 / Question 1]: Such point-level evaluation cannot guarantee that the corresponding reasoning is actually coherent in the report. […] I wonder whether the issue mentioned in Weakness 1 actually occurs in the benchmark cases and whether it affects the evaluation scores.

Thank you for your question. Independent rubric judgments are the inputs to our depth evaluation, while the aggregation preserves their structural relationships. To address your concern that “both a leaf-node rubric and its parent-node rubric may be satisfied by content appearing in different sections of the report,” we also examined the locations of the supporting evidence for consecutive satisfied rubrics to assess their textual proximity. Our analysis found this pattern relatively uncommon in the sampled cases.

We analyzed a selected set of 500 sampled reports. For groups of consecutive rubrics originally judged satisfied, we used Qwen3.8 Flash to locate supporting evidence sentences and checked whether they appeared in the same main body section. The rates below are averaged equally across reports.

Covered chainEvidence in the same main body section
1 hop: two satisfied parent–child rubrics87.12%
2 hops: three consecutive satisfied rubrics87.92%

In this subset, evidence for both chain lengths is predominantly located within the same main body section. We further inspected examples where the supporting evidence appeared in different sections and found that, in some cases, knowledge points in a coherent logical chain do not necessarily have to appear consecutively in the text. Reports may devote separate sections to different explanatory steps, analytical dimensions, or historical stages, or introduce a phenomenon in one section before tracing its origins in another. In such cases, mechanisms, supporting evidence, sources, or applicability conditions can connect explanations across sections and sustain meaningful progression.

Appendix E.4 provides a concrete example of consecutive parent–child rubrics satisfied across different report sections. We briefly describe this example here.

The research query is centered on Roguelike, a video game genre. The selected chain is shown below, with entities as nodes and rubrics as edges:

Roguelike
└─[R1]→ Rogue
         └─[R5]→ Hack
IDRubric requirement
R1The 1980 game Rogue is considered the forerunner and namesake of the roguelike genre.
R5Rogue inspired the development of Hack.

For R1→R5, evidence spans different sections while maintaining a clear historical reasoning coherence: R1 is satisfied in the section that describes how Rogue was developed and widely distributed, explaining how it became the genre’s defining model. R5 is satisfied in the subsequent section that discusses early roguelike variants and their developers, describing how Hack’s creator attempted to recreate Rogue from memory. The discussion progresses from establishing Rogue’s foundational role to showing how its design influenced a specific successor, preserving the continuous reasoning across the section boundary and thus demonstrating meaningful depth.

Further details are also provided in our separate response to [Weakness 2 / Question 2].

AuthorsOpenReview ↗

Rebuttal [2/3]

[Weakness 2 / Question 2]: Could the authors provide more examples of the actual reports generated by DR agents and their corresponding rubric satisfaction results?

Thank you for your suggestion. We have added a concrete report–rubric example in Appendix E.4 (Example Showcase). We briefly describe the example and its rubric satisfaction results below.

The root entity is Roguelike, a video game genre typically featuring procedurally generated dungeons and permadeath. The query asks for an analysis of the genre's origins, covering early games, technical constraints, tabletop influences, and distribution. We examine o3 Deep Research's report.

The selected relations are shown below, with entities as nodes and rubrics as edges (✓: satisfied; ✗: unmet). R10–R11 provide the path context for R12.

Roguelike
├─[R1 ✓]→ Rogue
│          ├─[R2 ✓]→ Permadeath
│          │          ├─[R3 ✓]→ Save scumming
│          │          └─[R4 ✓]→ Tabletop games
│          └─[R5 ✓]→ Hack
│                     └─[R6 ✗]→ Porting
└─[R10 ✓]→ Beneath Apple Manor
             └─[R11 ✓]→ Procedural generation
                          └─[R12 ✗]→ Random seed

Shared prefix omitted: “The report explicitly states or clearly paraphrases that”.

ID / SatisfiedRubric requirement
R1 ✓The 1980 game Rogue is considered the forerunner and namesake of the roguelike genre.
R2 ✓Rogue implements permadeath, meaning a character that loses all health is dead.
R3 ✓Players may circumvent permadeath by backing up and restoring savefiles, a tactic called 'save scumming' that is considered cheating.
R5 ✓Rogue inspired the development of Hack.
R6 ✗Hack was ported to other platforms, with some later ports using graphical tiles instead of ASCII characters.
R12 ✗Procedural generation used predefined seed values (map seeds) to generate very large game worlds.

Depth and continuity along knowledge chains. Along R1→R2→R3, the report progresses from Rogue’s role as the genre’s forerunner and namesake to permadeath, then to ways of circumventing and enforcing this mechanic. The following two consecutive sentences from the report support rubrics R2 and R3, respectively:

In Rogue, death was easy to come by and meant starting over from scratch, which forced players to focus on mastering the game’s systems rather than any single lucky run. (Notably, when some crafty students found they could copy save files to cheat death, Rogue’s authors changed the code to delete the save upon loading, enforcing permadeath.)

The next sentence adds the D&D connection (R4). Together, R2–R4 show progressively more specific coverage within one paragraph, with the intermediate relations covered.

Informative omissions. For R6, the report discusses graphical tiles for Beneath Apple Manor, not Hack, leaving the platform and display relation uncovered. For R12, it describes randomized dungeons without explaining the seed mechanism, a further technical extension. Despite being unmet, these rubrics remain valuable for the task, helping distinguish the specificity and completeness of reports’ knowledge coverage.

For R1→R5, evidence spans different sections while maintaining a clear historical connection: R1 is satisfied in the section that describes how Rogue was developed and widely distributed, explaining how it became the genre’s defining model. R5 is satisfied in the subsequent section that discusses early roguelike variants and their developers, describing how Hack’s creator attempted to recreate Rogue from memory. The discussion progresses from establishing Rogue’s foundational role to showing how its design influenced a specific successor, preserving the logical connection across the section boundary.

We sincerely appreciate your suggestion, which helped us further refine our paper.

AuthorsOpenReview ↗

Rebuttal [3/3]

[Weakness 3]: For example, a report may reach valid conclusions through alternative reasoning paths or different evidence that are not included in the rubric tree.

Thank you for your question. The benchmark's reference trees and rubrics are constructed from Wikipedia, and agents in our local research framework retrieve evidence over a fixed Wikipedia snapshot. Queries and rubrics originate from the same selected subgraph. Together, these components have already established a closed-loop evaluation within the local research framework, connecting construction, retrieval, and scoring through a common knowledge source.

Regarding the alternative reasoning paths or evidence you mention, it is important to first recognize that a further challenge concerns the completeness of the rubric tree’s coverage. For open-ended research, a finite reference cannot reasonably be expected to enumerate every valid research direction or cover all potentially relevant sources across the continually expanding open web. For example, AutoResearchBench [1] acknowledges that exhaustive answer sets remain difficult even with rigorous verification, while DeepResearch Bench [2] notes the difficulty of establishing a definitive standard answer for complex research queries. These works likewise ground evaluation in reference answers constructed from a finite body of source material, or in model-generated references, while acknowledging the difficulty of establishing exhaustive coverage.

More importantly, however, our approach to reference construction and structured evaluation is general in design and can be extended to other knowledge sources. The alternative reasoning paths or evidence you mention may draw on academic papers, industry reports, government records, or other sources. If we had sufficient access to these materials, along with the resources and permissions needed to process and reuse them, we could incorporate the relevant entities and relationships as new tree nodes and branches, yielding more comprehensive reference trees and rubrics.

For the current benchmark, we chose Wikipedia as one of the most readily accessible, openly licensed encyclopedic resources, offering public data dumps and clear reuse terms that facilitate large-scale benchmark construction and reproducible local retrieval. Wiki Live Challenge [3] and FreshWiki [4] also use Wikipedia articles as references, while TRQA’s Wiki subset [5] pairs structured knowledge with a Wikipedia corpus. These precedents also support our choice of Wikipedia as a reasonable foundation for benchmark construction and evaluation.

Ref:

[1] Xiong et al. AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery.

[2] Du et al. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents.

[3] Wang et al. Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles.

[4] Shao et al. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models.

[5] Rafiee et al. Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents.


Thank you again for the thorough review. We hope our responses address your concerns. If you have any further questions, we would be happy to answer them.

Sincerely,
DRBench Pro Authors

Reviewer_AqUwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DRBench Pro to evaluate how thoroughly deep research reports cover structured knowledge. The authors construct entity-relation trees from Wikipedia and sample research scopes from them, producing 144 questions and 44,102 hidden scoring criteria across 24 domains and 48 root entities. For each question, ten branches beneath the root entity and their descendant relations are retained. A judge model assesses whether the report covers each relation. These judgments are then aggregated over the tree to calculate breadth and depth, with their geometric mean serving as the overall coverage score. The evaluation covers 16 systems and models, including deep research systems, native search models, and backbone models integrated into a local research framework. The authors also report human–judge agreement and analyses of metric stability.

Critical strengths
  1. The paper is clearly presented and proposes a knowledge coverage evaluation method for deep research agents that considers both depth and breadth.

  2. The benchmark is built from reliable Wikipedia pages, and its construction process is easy to scale.

Critical weaknesses
  1. The authors introduce the concept of “depth coverage,” but do not sufficiently establish whether each question tells the agent which child nodes to explore in depth and why covering those nodes is necessary. The human review also does not appear to address this issue. If the rubric includes relations that cannot reasonably be anticipated from the question, a low score may reflect an unspecified scope rather than insufficient research depth.

  2. The questions were constructed using Claude Opus 4.8 and may therefore reflect that model’s research preferences to some extent. Claude Opus 5 ranks highest among the Native Search LLMs.

  3. The label “Ours: Backbone LLMs under the local research framework” in Table 2 is inappropriate, given that the paper does not propose a new deep research framework.

  4. Although the paper evaluates deep research, only three of the evaluated systems are deep research systems; the others are search agents equipped with standard search tools. The evaluation should include open-source deep research harnesses, such as LangChain Open Deep Research. The reported results show that o3 Deep Research has a clear score advantage over more advanced models running under other harnesses, particularly in depth. This also suggests that the evaluated harnesses may not have been designed specifically for deep research.

  5. Most deep research agents likely use a multi-stage design: they search extensively for relevant literature during retrieval, then select valuable material and compose the report during writing. This process can improve the recall of retrieved facts. However, empirical results show that agents may discard many of those facts during writing, reflecting their own preferences about which knowledge to include. Applying the proposed method to study different models’ breadth and depth preferences during retrieval and writing would be valuable.

  6. The search process does not exclude the root entity’s own page. Moreover, the question about Lew Grade appears to provide an overly explicit search path, allowing the model to complete the research simply by following the question from the root entity page to its child nodes. This substantially narrows the search space to a relatively clean environment. The evaluation therefore seems closer to measuring whether a model can cover content selected by the benchmark’s creators within a preselected, closed knowledge base that closely matches the question, rather than fully measuring autonomous research ability on the open web.

Questions for authors

See the weaknesses above.

5 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Dear Reviewer AqUw,

Thank you for the effort and time you dedicated to reviewing our paper, and for your recognition of the contribution of our benchmark and its evaluation approach. We have greatly benefited from your valuable feedback. Below are our responses to the questions you pointed out:


[Weakness 1]: If the rubric includes relations that cannot reasonably be anticipated from the question, a low score may reflect an unspecified scope rather than insufficient research depth.

Thank you for your question. Research directions are communicated through the query’s analytical goals, while specific entities remain to be discovered. Queries and rubrics derive from the same sampled subgraph, whose branches represent complementary research directions around the root topic. Depth measures coverage along these reference chains. We reviewed 144 queries for research value, public answerability, and domain match (Table 4). Accordingly, the intended coverage requirements concern relations that substantively support the query’s stated analytical goals, rather than additional research objectives left unspecified in the question.

[Weakness 2]: The questions were constructed using Claude Opus 4.8 and may therefore reflect that model’s research preferences to some extent.

Thank you for raising this question. While several related benchmarks, such as DeepResearchEval [1] (§5.3) and DeepResearch Bench II [2] (§4.3.2), only assess robustness across judge models, we additionally examine potential sensitivity to the model used to construct the tasks and rubrics. To this end, we conducted a supplementary experiment by reconstructing reference graphs, queries, and rubrics with Kimi K2.6. On 10 randomly sampled cases, we evaluated Claude Opus 5 alongside three Native Search LLMs with relatively close performance to Claude in our original evaluation. We compared the models’ results on the reconstructed benchmark with their results on the original benchmark constructed by Claude Opus 4.8.

To quantify Claude’s relative advantage over each opponent, we report the mean standardized pairwise gap in Overall scores, computed separately for each benchmark version as Gapm=110∑i=110Si,C−Si,mσi\mathrm{Gap}_m=\frac{1}{10}\sum_{i=1}^{10}\frac{S_{i,C}-S_{i,m}}{\sigma_i}, where Si,CS_{i,C} and Si,mS_{i,m} are the Overall scores of Claude and opponent mm on case ii, and σi\sigma_i is the population standard deviation of the four model scores within that case.

The table below summarizes sensitivity to the benchmark construction model. Ranks are computed among the four models within each case and averaged across cases. Positive gaps favor Claude.

Evaluated modelOriginal Overall (%)Kimi-built Overall (%)Original rank ↓Kimi-built rank ↓Original gapKimi-built gap
Claude Opus 556.8440.201.21.8——
Grok 4.551.2537.632.51.9+0.869+0.336
GPT-5.6 Sol46.6534.213.02.7+1.546+0.807
Sonar Reasoning Pro44.0925.843.33.6+1.944+1.804

Under Kimi-based construction, Claude retains the highest mean Overall score and appreciable positive standardized gaps against all three opponents in the sampled cases, although its relative advantage narrows. Taken together, these findings suggest that Claude’s observed advantage can persist when the benchmark is constructed by a different model family, although the choice of construction model may still play a role. This analysis is also included in §4.7 of the revised manuscript.

Once again, we thank you for raising this point, which has helped us further improve our paper.

Ref:

[1] Wang et al. DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation.

[2] Li et al. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report.

AuthorsOpenReview ↗

Rebuttal [2/3]

[Weakness 3]: The label “Ours: Backbone LLMs under the local research framework” in Table 2 is inappropriate, given that the paper does not propose a new deep research framework.

Thank you for pointing this out. We have removed “Ours:” from Table 2 and relabeled this group “Backbone LLMs under the local research framework.” The framework serves as a controlled evaluation setting for comparing backbone LLMs, while our focus is on the benchmark and structured knowledge coverage evaluation.

[Weakness 4]: The evaluation should include open-source deep research harnesses, such as LangChain Open Deep Research.

Thank you for your suggestion. We have so far evaluated LangChain Open Deep Research (ODR) with DeepSeek V4 Flash, using Tavily for web search, on 24 sampled cases. The table summarizes average results for representative configurations. Both LangChain ODR and the local research framework use DeepSeek V4 Flash.

ConfigurationOverall (%)Breadth (%)Depth (%)search queriesModel-callReport words
LangChain ODR57.1151.9863.29133.67693.297,130
Local research framework45.4345.4946.489.8828.041,437
o3 Deep Research52.1547.5457.78——5,344

ODR achieves Overall and Depth scores 11.67 and 16.81 percentage points higher than the local configuration, respectively, and also exceeds o3 Deep Research on this subset. Its larger exploration budget, iterative supervision, and evidence synthesis may contribute to these differences. This supports including dedicated open-source harnesses and motivates studying how research budgets and workflows translate into coverage, alongside backbone capability. A more detailed presentation is provided in Appendix D.5.

As future work, we plan to extend this evaluation to the full benchmark, incorporating additional open-source deep research harnesses and backbone models under controlled research budgets.

Once again, we thank you for raising this point, which has helped us further improve our paper.

AuthorsOpenReview ↗

Rebuttal [3/3]

[Weakness 5]: Applying the proposed method to study different models’ breadth and depth preferences during retrieval and writing would be valuable.

Thank you for your suggestion. We evaluated retrieval and report coverage on 24 cases for the seven backbone LLMs evaluated under the local research framework. At the retrieval stage, we collected the retrieved materials and measured the model's knowledge coverage at this stage in terms of Breadth and Depth. At the writing stage, we evaluated the final reports using the same method.

The table summarizes coverage at both stages and knowledge retention. Arrows indicate retrieval → report. Retention measures the proportion of fully covered retrieval rubrics retained in the final reports. Models are ordered by their final-report Overall scores in descending order.

ModelBreadth (%)Depth (%)Knowledge retentionRetention by hop distance (1 / 2 / 3)
Qwen3.8 Max49.36 → 44.3554.64 → 50.1665.6%88.1% / 63.0% / 45.8%
GLM 5.255.04 → 45.6557.80 → 46.4052.0%84.1% / 49.6% / 30.1%
MiniMax M353.46 → 42.7752.91 → 45.6856.7%79.0% / 55.2% / 35.4%
DeepSeek V4 Flash49.38 → 44.3954.75 → 44.2056.5%89.8% / 49.6% / 32.8%
DeepSeek V4 Pro43.33 → 34.8547.59 → 43.9953.7%74.7% / 52.0% / 34.0%
Qwen3.7 Plus52.87 → 33.7553.65 → 37.6936.4%66.3% / 28.8% / 16.0%
Gemma 4 31B43.66 → 30.0442.18 → 34.1850.7%66.5% / 48.4% / 22.9%

Overall, 47.0% of fully covered retrieval rubric instances are no longer fully covered in reports, and knowledge retention decreases as hop distance increases. Models also differ in what they preserve: DeepSeek Pro preserves more of its Depth than Breadth score (92.4% vs. 80.4%), suggesting more focused coverage along selected chains. Flash better preserves Breadth (89.9% vs. 80.7% for Depth), consistent with retaining topical range while compressing deeper content. Qwen3.8 Max maintains high score retention in both dimensions (89.8%/91.8%) and retains the most retrieved knowledge (65.6%), showing stronger coverage preservation during synthesis.

Evaluating retrieval and reporting separately distinguishes retrieval gaps from omissions of already retrieved knowledge in the final report. This offers valuable guidance for improving retrieval and knowledge retention during synthesis. We sincerely appreciate your suggestion, which helped us further refine our paper. Further details are provided in Appendix D.4.

[Weakness 6]: The search process does not exclude the root entity’s own page. Moreover, the question about Lew Grade appears to provide an overly explicit search path, allowing the model to complete the research simply by following the question from the root entity page to its child nodes.

Thank you for your question. The root page is a natural source for researching the named topic. The local framework searches a shared Wikipedia snapshot, rather than a corpus restricted to the selected subgraph. Agents receive only the query; reference trees and rubrics remain hidden.

The Lew Grade query states analytical goals rather than a sequence of Wikipedia links. Our 24-case audit (§4.5, Table 5) confirmed eight entity-name leaks across 7,385 query–entity comparisons (0.108%), with none among 240 first-hop entities. This supports limited direct disclosure of hidden entities.

The controlled corpus is a deliberate choice for reproducible evaluation of report breadth and depth. Our focus is structured knowledge coverage, rather than a comprehensive assessment of autonomous research on the open web.


Thank you again for the thorough review. We hope our responses address your concerns. If you have any further questions, we would be happy to answer them.

Sincerely,
DRBench Pro Authors

Reviewer_ibQ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DRBench Pro, a benchmark evaluating how well deep research reports cover a structured body of knowledge. Building from Wikipedia, the benchmark contains 144 cases across 24 domains, with 44,102 rubrics. The evaluation produces separate breadth and depth scores, which measure coverage across research directions and along knowledge chains. Their geometric mean gives the overall score. The authors evaluate 16 agent configurations. They also assess query quality, test sensitivity to metric parameters, and compare automated judgments with 3,695 human-reviewed rubric judgments from 12 reports. The selected judge achieves a binary Macro-F1 of 0.893. The main contribution is an automated benchmark construction pipeline with separate structural measures of report coverage.

Critical strengths
  1. The separate breadth and depth scores are useful for diagnosing coverage gaps. This breakdown helps researchers identify specific weaknesses in generated reports and guide changes to research workflows. The definition of depth as knowledge-chain coverage in the final report also gives the metric a clear scope.
  2. The benchmark is traceable, and the automated judge receives meaningful validation. Each rubric is linked to a source relation and supporting Wikipedia evidence. This makes individual scoring requirements easy to inspect. The appendix provides detailed prompts and formulas, which help readers understand how the benchmark was built. The human–judge study provides useful evidence of agreement with the reviewed human labels. The shared local workflow for seven backbones also provides a controlled setting for comparing models. These features support the benchmark’s value as a resource for studying report coverage.
Critical weaknesses
  1. The reference content needs a direct check against the visible research question. Each task retains complete descendant branches, and every retained edge becomes a rubric. Non-root expansion selects links for their relevance to the current source entity. Rubric generation then uses the source and target page summaries and the relationship evidence. An independent audit against the final query is needed to establish whether these requirements are relevant, appropriately scoped, and reasonable to expect in an answer. All construction stages use Claude Opus 4.8. Meanwhile, Claude Opus 5 leads in 23 of 24 domains and achieves 57.04% overall coverage. These results make potential model-family preferences in task and rubric construction worth examining. The reference trees also depend entirely on Wikipedia. Rubrics can be satisfied by information from any source. Professional research may follow relevant paths through academic papers, industry reports, or government records that extend beyond the selected tree. Such paths receive limited visibility in the coverage score. An expert review of both selected and omitted research directions would help establish the benchmark’s scope. Table 4 supports query quality; the alignment between each query and its full set of hidden requirements also needs direct validation.

  2. The interpretation and added value of the scores need stronger empirical support. The metric rewards coverage of reference relations. Relevance and selectivity across the full report need a separate check. Table 12 shows a strong association between average report length and overall coverage across the seven local backbones. Qwen3.8 Max averages 2,301.6 words, Qwen3.7 Plus 859.0 words, and Gemma 4 31B 629.9 words. This makes length an important factor to control. Length statistics for all 16 configurations and comparisons under similar output budgets would help interpret the main rankings. The human reference also needs further documentation. Initial inter-annotator agreement, the number of independent labels per item, and the handling of disagreements would help readers assess annotation reliability. These details matter when interpreting the judge’s 0.893 Macro-F1 against the final reviewed labels. The methodological contribution also needs a direct comparison with related scoring approaches. For example, "Super Research" uses topology-derived weights for report coverage and checks evidence-support paths; "From Simple QA to Deep Research" propagates weights through a DAG to score final responses. The introduction should describe this shared ground more precisely. Comparisons with flat coverage and structural baselines, using fixed reports and rubric judgments, would help establish their value. Independent human assessments of report-level breadth and depth would provide a useful reference for this comparison.

Questions for authors

As listed in Weaknesses.

6 comments
AuthorsOpenReview ↗

Rebuttal [1/4]

Dear Reviewer ibQ8,

Thank you for the effort and time you dedicated to reviewing our paper, and for your recognition of the contribution of our benchmark and its evaluation approach. We have greatly benefited from your valuable feedback. We address each point in the two weaknesses below.


[Weakness 1a]: These results make potential model-family preferences in task and rubric construction worth examining.

Thank you for raising this question. While several related benchmarks, such as DeepResearchEval [1] (§5.3) and DeepResearch Bench II [2] (§4.3.2), only assess robustness across judge models, we additionally examine potential sensitivity to the model used to construct the tasks and rubrics. To this end, we conducted a supplementary experiment by reconstructing reference graphs, queries, and rubrics with Kimi K2.6. On 10 randomly sampled cases, we evaluated Claude Opus 5 alongside three Native Search LLMs with relatively close performance to Claude in our original evaluation. We compared the models’ results on the reconstructed benchmark with their results on the original benchmark constructed by Claude Opus 4.8.

To quantify Claude’s relative advantage over each opponent, we report the mean standardized pairwise gap in Overall scores, computed separately for each benchmark version as Gapm=110∑i=110Si,C−Si,mσi\mathrm{Gap}_m=\frac{1}{10}\sum_{i=1}^{10}\frac{S_{i,C}-S_{i,m}}{\sigma_i}, where Si,CS_{i,C} and Si,mS_{i,m} are the Overall scores of Claude and opponent mm on case ii, and σi\sigma_i is the population standard deviation of the four model scores within that case.

The table below summarizes sensitivity to the benchmark construction model. Ranks are computed among the four models within each case and averaged across cases. Positive gaps favor Claude.

Evaluated modelOriginal Overall (%)Kimi-built Overall (%)Original rank ↓Kimi-built rank ↓Original gapKimi-built gap
Claude Opus 556.8440.201.21.8——
Grok 4.551.2537.632.51.9+0.869+0.336
GPT-5.6 Sol46.6534.213.02.7+1.546+0.807
Sonar Reasoning Pro44.0925.843.33.6+1.944+1.804

Under Kimi-based construction, Claude retains the highest mean Overall score and appreciable positive standardized gaps against all three opponents in the sampled cases, although its relative advantage narrows. Taken together, these findings suggest that Claude’s observed advantage can persist when the benchmark is constructed by a different model family, although the choice of construction model may still play a role. This analysis is also included in §4.7 of the revised manuscript.

Once again, we thank you for raising this point, which has helped us further improve our paper.

Ref:

[1] Wang et al. DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation.

[2] Li et al. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report.

AuthorsOpenReview ↗

Rebuttal [2/4]

[Weakness 1b]: An independent audit against the final query is needed to establish whether these requirements are relevant, appropriately scoped, and reasonable to expect in an answer. […] Professional research may follow relevant paths through academic papers, industry reports, or government records that extend beyond the selected tree. […] An expert review of both selected and omitted research directions would help establish the benchmark’s scope.

Thank you for your question. The benchmark's reference trees and rubrics are constructed from Wikipedia, and agents in our local research framework retrieve evidence over a fixed Wikipedia snapshot. Queries and rubrics originate from the same selected subgraph. This shared research scope has already provided a natural basis for query–rubric alignment. Together, these components have already established a closed-loop evaluation within the local research framework, connecting construction, retrieval, and scoring through a common knowledge source.

Table 4 already reports our manual assessment of all 144 queries for research value, public answerability, and query–domain match. Regarding your suggestion to review selected and omitted research directions, a further challenge concerns the completeness of the reference scope. For open-ended research, a finite reference cannot reasonably be expected to enumerate every valid research direction or cover all potentially relevant sources across the continually expanding open web. AutoResearchBench [1] acknowledges that exhaustive answer sets remain difficult even with rigorous verification, while DeepResearch Bench [2] notes the difficulty of establishing a definitive standard answer for complex research queries.

More importantly, however, our approach to reference construction and structured evaluation is general in design and can be extended to other knowledge sources. The additional research directions you mention may draw on academic papers, industry reports, government records, or other sources. If we had sufficient access to these materials, along with the resources and permissions needed to process and reuse them, we could incorporate the relevant entities and relationships as new tree nodes and branches, yielding more comprehensive reference trees and rubrics.

For the current benchmark, we chose Wikipedia as one of the most readily accessible, openly licensed encyclopedic resources, offering public data dumps and clear reuse terms that facilitate large-scale benchmark construction and reproducible local retrieval. Wiki Live Challenge [3] and FreshWiki [4] also use Wikipedia articles as references, while TRQA’s Wiki subset [5] pairs structured knowledge with a Wikipedia corpus. These precedents also support our choice of Wikipedia as a reasonable foundation for benchmark construction and evaluation.

Ref:

[1] Xiong et al. AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery.

[2] Du et al. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents.

[3] Wang et al. Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles.

[4] Shao et al. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models.

[5] Rafiee et al. Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents.

AuthorsOpenReview ↗

Rebuttal [3/4]

[Weakness 2a]: Length statistics for all 16 configurations and comparisons under similar output budgets would help interpret the main rankings.

Thank you for raising this point. We added length statistics for all 16 configurations to Appendix D.2 of the revised manuscript. However, we believe that more comprehensive knowledge coverage can require more space. A report presenting more substantive facts and relations satisfies more rubrics when those additions match the reference knowledge. The resulting increase in coverage is consistent with our evaluation objective; length itself earns no credit.

Existing benchmarks do not enforce a common report word count across systems in their main comparisons. ResearchRubrics [1] (§4.2–4.5) notes that longer reports can satisfy more criteria, while acknowledging potential verbosity bias. LiveResearchBench [2] (§5) directly uses particularly generous output budgets.

For open-ended deep research, forcing reports to an identical length adds compression and content-selection constraints that change the evaluation target. Our primary setting therefore allows systems to develop their reports within their own configured research budgets.

To address your concern about comparable output budgets, we additionally reran all seven local backbones on 24 sampled cases, one from each of the 24 domains, retaining research settings while targeting 1,200 words. The table below reports means across the 24 cases.

BackboneOriginal Overall (%)Target-length Overall (%)Original wordsTarget-length words
Qwen3.8 Max49.1143.312,251.91,198.4
MiniMax M345.1341.941,899.21,349.7
GLM 5.245.1644.301,622.01,496.4
DeepSeek V4 Flash43.2038.191,448.01,224.7
DeepSeek V4 Pro36.8638.071,307.81,239.0
Qwen3.7 Plus35.8233.41878.31,163.4
Gemma 4 31B32.0934.01642.71,035.0

It can be seen that differences in knowledge coverage remain under a common length target, with some changes in relative ordering. The same three models remain at the top, although their ordering changes. For example, Qwen3.8 Max and DeepSeek V4 Pro have similar lengths (1,198 vs. 1,239 words) but differ by 5.24 Overall percentage points.

Although we explicitly specified a common length target, adherence varied across models: MiniMax and GLM tended to exceed it, whereas Gemma tended to fall short. Nevertheless, average report lengths became substantially closer, providing a more comparable basis for this supplementary evaluation.

Ref:

[1] Sharma et al. ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents.

[2] Wang et al. LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild.

[Weakness 2b]: Initial inter-annotator agreement, the number of independent labels per item, and the handling of disagreements would help readers assess annotation reliability.

Thank you for your question. In Appendix I of our original submission, we already described the annotators, three-category labeling criteria (Satisfied, Partially Satisfied, and Not Satisfied), and second human review used to establish the human reference. Specifically, we reported that the evaluation involved 10 PhD-level annotators and 3,695 report–rubric pairs from 12 reports, each representing a different domain.

We added further protocol details to Appendix I.1 of the revised manuscript, and we briefly describe them here. The protocol specifies two independent initial labels per report–rubric pair, with annotators having no access to one another’s labels or the LLM judges’ verdicts. Initial inter-annotator agreement was Cohen’s κ = 0.841, computed before adjudication over all 3,695 report–rubric pairs using the original three-category labels. A third annotator adjudicates disagreements using the relevant report passages and the same criteria, producing the final human reference. The reported 0.893 Macro-F1 measures Gemini’s agreement with this final reference, averaged equally across 12 reports.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal & Discussion phases

Dear Reviewers, AC, SAC, and PC,

We sincerely appreciate the time and effort you devoted to reviewing our work, as well as the insightful feedback you provided, which has significantly strengthened our paper. Below, we summarize the progress so far.


We are encouraged to see that the reviewers recognized several key strengths of our work, including:

  • Well-motivated benchmark: Evaluating both breadth and depth reflects the goals of deep research and provides a new way to assess reports. Separate scores help diagnose coverage gaps, and the definition of depth as knowledge-chain coverage also gives the metric a clear scope. (Reviewers ibQ8, AqUw, eean)
  • Scalable benchmark construction from authoritative sources: Our pipeline generates research tasks from Wikipedia and is easy to scale. (Reviewers AqUw, eean)
  • Traceable evaluation rubrics: Each rubric links to a source relation and supporting Wikipedia evidence. (Reviewer ibQ8)
  • Human validation of the automated judge: Annotations provide an independent reference for evaluating rubric judgments. (Reviewer ibQ8)
  • Controlled comparison across seven backbones: Seven models use the shared local research workflow. (Reviewer ibQ8)

During the rebuttal phase, we conducted additional experiments or supplemented relevant analyses in the revision to address the reviewers' concerns:

  • To test sensitivity to the construction model, we rebuilt reference graphs, queries, and rubrics with Kimi K2.6. On 10 sampled cases, Claude Opus 5 still retained the highest Overall score, while its standardized lead was smaller than on the original benchmark. This suggests that Claude’s lead can persist across construction models, though its magnitude may vary (Section 4.7). (Reviewers ibQ8, AqUw)
  • Across seven backbones and 24 cases, we evaluated coverage both in retrieved materials at the retrieval stage and in final reports at the writing stage. Models differ in what they preserve: during writing, DeepSeek V4 Pro preferentially preserved depth, while DeepSeek V4 Flash favored breadth. Retention declined with hop distance (Appendix D.4). (Reviewer AqUw)
  • We reran seven backbones in our local research framework on 24 cases, with a common 1,200-word target. The same three models led in Overall coverage, though their order changed (Appendix D.3). (Reviewer ibQ8)
  • We evaluated LangChain Open Deep Research (ODR) with DeepSeek V4 Flash as its backbone on 24 cases. ODR achieved 57.11% mean Overall coverage. It issued more search queries and produced longer reports than our local framework (Appendix D.5). (Reviewer AqUw)
  • We found that, within the 500 reports examined, supporting evidence for consecutive satisfied rubrics usually appeared in the same section. We also reviewed examples to illustrate how a reasoning chain can remain coherent across section boundaries (Appendix E.4). (Reviewer eean)
  • We provided an example including a report and its corresponding rubric satisfaction results to illustrate how the evaluation works in practice (Appendix E.4). (Reviewer eean)

We also clarified several key points to address the reviewers' concerns:

  • We explained why we chose Wikipedia as a common source for benchmark construction, retrieval, and evaluation. We have thus established a closed-loop evaluation within our local research framework. We also discussed the coverage limits of a finite reference in open-ended research and clarified that our approach is general in design and can be extended to other knowledge sources. (Reviewers ibQ8, eean)
  • We provided further details on the human reference labels and the annotation protocol. We reported two independent labels per item, initial inter-annotator agreement, and the adjudication procedure (Appendix I.1). (Reviewer ibQ8)
  • We clarified how each query communicates the intended research directions while leaving specific entities for the agent to discover. We also explained how the reference rubrics relate to the query’s stated goals and define the scope of the coverage evaluation. (Reviewers ibQ8, AqUw)
  • We clarified how our evaluation builds on prior work that uses task-specific structured references and how our approach extends this line of work by designing distinct metrics for breadth and depth. (Reviewer ibQ8)

During the discussion, Reviewer eean acknowledged that our rebuttal and additional evidence partially addressed their concerns and raised their rating from 2 to 3. We regret that we have not yet had the opportunity to discuss our responses with Reviewers ibQ8 and AqUw.


Once again, we sincerely thank all reviewers for their recognition and valuable suggestions. We believe that our responses address the raised concerns effectively, and we hope the program committee (reviewers, ACs, SACs, PCs) will take them into account in the next phases.

Sincerely,
DRBench Pro Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.