- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
LifeSearch evaluates 12 deep research (DR) LLM agents, including GPT-5.6 Sol, Opus 5, and Qwen3.8, on 203 human-annotated tasks in Chinese and English. The benchmark covers 7 primary research domains and compares DR models under a shared evaluation framework and rubrics, assessing their ability to (1) collect relevant information through web search and information retrieval and (2) synthesize information into a research report that explicitly addresses the user’s needs. An LLM judge, Gemini 3.1 Pro, scores the research reports generated by DR agents using the evaluation rubric written by human experts. The highest-scoring DR agent, GPT-5.6 Sol, achieves 63.44% weighted rubric satisfaction.
Critical strengths
Perhaps the strongest offering of the benchmark is its bilingual coverage in English and Chinese, together with human-annotated tasks and evaluation rubrics written by human domain experts. The annotation effort is substantial, and the 7 primary task domains cover a range of practical research needs, including company and competitor research, financial market analysis, and university tuition and admissions comparisons.
Critical weaknesses
-
Lack of a clearly established scientific contribution. Human-annotated datasets are useful, but the paper needs to explain what new knowledge LifeSearch contributes beyond adding another evaluation benchmark. Existing evaluation benchmark such as ResearchRubrics already uses expert-written evaluation criteria, while LiveResearchBench and DRACO cover overlapping real-world research settings. Although Table 1 highlights several distinctive dataset characteristics, the empirical analysis does not sufficiently demonstrate whether these characteristics reveal new capability gaps, failure modes, or model-ranking differences that existing benchmarks fail to capture. More direct comparisons with related benchmarks, or analyses isolating the contributions of bilingual coverage, real-world task settings, and numerically grounded rubrics, would strengthen the claimed novelty. The paper should clarify whether LifeSearch’s contribution lies in a new evaluation paradigm, new ML metrics or statistical estimators, or improved measurement of previously underexamined capabilities of DR agents. The empirical setup should be more focused on demonstrating what LifeSearch enables researchers to measure or learn that other evaluation benchmarks do not adequately capture.
-
The evaluation scoring rubrics need stronger justification of their validity, generalizability, and robustness. An LLM judge uses the evaluation rubrics to assess whether a DR agent’s generated report meets specified task requirements. How the evaluation rubrics can reliably estimate the broader research capabilities of DR agents—including factual accuracy and the handling of conflicting evidence—requires clearer conceptual justification and further empirical validation. Also, about 40% of the criteria involve implicit requirements: expectations that annotators consider reasonably implied by a task but that are not explicitly requested. Examples include presenting university comparisons in a table or providing source URLs for government statistics. The selection and weighting of these requirements involve annotation choices whose influence on the evaluation results should be examined. Although all models are evaluated on the same tasks, the paper should assess how the evaluation results generalize under different settings such as equivalent task rephrasings, different weightage in the scoring rubrics, and different choices of task domains. This would clarify how far the reported model comparisons generalize beyond the particular tasks and criteria selected in the rubrics.
-
The evaluation does not sufficiently distinguish the underlying capabilities of deep research agents. The paper identifies search planning, information synthesis, conflict resolution, and uncertainty handling as important capabilities of deep research agents. However, its primary evaluation metric aggregates satisfaction of final-report requirements rather than directly measuring these capabilities separately. Although the paper analyzes search iterations and report length, these descriptive relationships do not establish which capabilities explain differences in model performance. Consequently, it is difficult to determine whether a low score reflects inadequate evidence retrieval, incorrect reasoning, failure to reconcile conflicting information, or simply incomplete presentation of otherwise correctly gathered information. A more systematic capability-level analysis, potentially using the existing rubric dimensions or annotated task requirements, would make the benchmark more diagnostically informative and strengthen its contribution beyond overall model ranking.
Questions for authors
1.Scientific contribution and differentiation: What specific research capabilities, failure modes, or model-performance differences does LifeSearch reveal that existing benchmarks such as ResearchRubrics, LiveResearchBench, and DRACO do not adequately capture? Could the authors provide comparative or targeted empirical analyses to substantiate the distinctive scientific contribution of LifeSearch?
2.Construct validity and capability measurement: Although the paper provides human agreement studies, to what extent do the rubric-based scores reliably measure the underlying capabilities of deep research agents, particularly factual accuracy, evidence synthesis, conflict resolution, and uncertainty handling? Could the authors provide capability-level analyses or additional validation beyond agreement with human judgments?
3.Robustness and generalizability: How sensitive are the reported model scores and rankings to rubric-design and task-selection choices? In particular, would the conclusions remain consistent when implicit requirements are excluded, alternative rubric weights are used, task-domain compositions are changed, or semantically equivalent task rephrasings are evaluated?
Thank you for the detailed response and the additional analysis conducted. The sensitivity checks and additional LLM-judge evaluation do address my concerns about the stability of the reported rankings under the settings described in the paper. However, the distinctive scientific contribution of LifeSearch remains insufficiently established, as it is not demonstrated which of LifeSearch's capabilities existing SOTA benchmarks fail to capture. The summary table of the comparison between LifeSearch and other SOTA models did not convincingly show the additional capability or gap that LifeSearch addresses but other SOTA models do not. The only differences seem to be increasing the number of domains covered to 7 and including bilingual tasks, but these are design choices rather than the inherent limitations of other SOTA models. It is almost certain that other SOTA benchmark models can also cover 7 or perhaps more domains if needed.
I appreciate the improvements provided in the rebuttals, but maintain my weak-reject recommendation because the contribution and the capability-measurement claims remain only partially supported.