acceptodds
Under review as a conference paper at ICLR 2027

LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity

Abstract

Deep research (DR) agents address open-ended information needs through multi-step reasoning, long-horizon web search, and evidence synthesis. However, existing benchmarks primarily focus on academic research or synthetic tasks, leaving DR agents’ capabilities in real-world life and professional productivity settings insufficiently evaluated. To bridge this gap, we introduce LifeSearch, a challenging benchmark comprising 203 Chinese and English tasks derived from real-user query seeds across 7 concrete life and productivity domains. Specially, we spend totally more than 4,500 hours of human annotation to construct over 4,000 expert-written, fine-grained rubrics, covering both report-content criteria for coverage and accuracy, as well as concrete criteria specifying values, units, dates, and URLs. We conduct extensive experiments on 12 advanced DR agents under a unified evaluation harness, systematically analyzing their performance patterns and failure modes. We find that even the best-performing frontier agent, GPT-5.6 Sol, achieves a score of only 63.4%, revealing the potential improvement on realistic life and professional DR tasks. We compare candidate LLM judges against human rubric labels to select the grading configuration. A human ranking study shows strong alignment between aggregate benchmark scores and human judgments, with Spearman's and Kendall's . Overall, LifeSearch provides a rigorous evaluation framework and diagnostic foundation for advancing DR agents toward real-world life and productivity applications.

Primary area:
datasets and benchmarks
Keywords:
LLM based Agent, Evaluation, Deep Research, Rubrics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GiUEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LifeSearch evaluates 12 deep research (DR) LLM agents, including GPT-5.6 Sol, Opus 5, and Qwen3.8, on 203 human-annotated tasks in Chinese and English. The benchmark covers 7 primary research domains and compares DR models under a shared evaluation framework and rubrics, assessing their ability to (1) collect relevant information through web search and information retrieval and (2) synthesize information into a research report that explicitly addresses the user’s needs. An LLM judge, Gemini 3.1 Pro, scores the research reports generated by DR agents using the evaluation rubric written by human experts. The highest-scoring DR agent, GPT-5.6 Sol, achieves 63.44% weighted rubric satisfaction.

Critical strengths

Perhaps the strongest offering of the benchmark is its bilingual coverage in English and Chinese, together with human-annotated tasks and evaluation rubrics written by human domain experts. The annotation effort is substantial, and the 7 primary task domains cover a range of practical research needs, including company and competitor research, financial market analysis, and university tuition and admissions comparisons.

Critical weaknesses
  1. Lack of a clearly established scientific contribution. Human-annotated datasets are useful, but the paper needs to explain what new knowledge LifeSearch contributes beyond adding another evaluation benchmark. Existing evaluation benchmark such as ResearchRubrics already uses expert-written evaluation criteria, while LiveResearchBench and DRACO cover overlapping real-world research settings. Although Table 1 highlights several distinctive dataset characteristics, the empirical analysis does not sufficiently demonstrate whether these characteristics reveal new capability gaps, failure modes, or model-ranking differences that existing benchmarks fail to capture. More direct comparisons with related benchmarks, or analyses isolating the contributions of bilingual coverage, real-world task settings, and numerically grounded rubrics, would strengthen the claimed novelty. The paper should clarify whether LifeSearch’s contribution lies in a new evaluation paradigm, new ML metrics or statistical estimators, or improved measurement of previously underexamined capabilities of DR agents. The empirical setup should be more focused on demonstrating what LifeSearch enables researchers to measure or learn that other evaluation benchmarks do not adequately capture.

  2. The evaluation scoring rubrics need stronger justification of their validity, generalizability, and robustness. An LLM judge uses the evaluation rubrics to assess whether a DR agent’s generated report meets specified task requirements. How the evaluation rubrics can reliably estimate the broader research capabilities of DR agents—including factual accuracy and the handling of conflicting evidence—requires clearer conceptual justification and further empirical validation. Also, about 40% of the criteria involve implicit requirements: expectations that annotators consider reasonably implied by a task but that are not explicitly requested. Examples include presenting university comparisons in a table or providing source URLs for government statistics. The selection and weighting of these requirements involve annotation choices whose influence on the evaluation results should be examined. Although all models are evaluated on the same tasks, the paper should assess how the evaluation results generalize under different settings such as equivalent task rephrasings, different weightage in the scoring rubrics, and different choices of task domains. This would clarify how far the reported model comparisons generalize beyond the particular tasks and criteria selected in the rubrics.

  3. The evaluation does not sufficiently distinguish the underlying capabilities of deep research agents. The paper identifies search planning, information synthesis, conflict resolution, and uncertainty handling as important capabilities of deep research agents. However, its primary evaluation metric aggregates satisfaction of final-report requirements rather than directly measuring these capabilities separately. Although the paper analyzes search iterations and report length, these descriptive relationships do not establish which capabilities explain differences in model performance. Consequently, it is difficult to determine whether a low score reflects inadequate evidence retrieval, incorrect reasoning, failure to reconcile conflicting information, or simply incomplete presentation of otherwise correctly gathered information. A more systematic capability-level analysis, potentially using the existing rubric dimensions or annotated task requirements, would make the benchmark more diagnostically informative and strengthen its contribution beyond overall model ranking.

Questions for authors

1.Scientific contribution and differentiation: What specific research capabilities, failure modes, or model-performance differences does LifeSearch reveal that existing benchmarks such as ResearchRubrics, LiveResearchBench, and DRACO do not adequately capture? Could the authors provide comparative or targeted empirical analyses to substantiate the distinctive scientific contribution of LifeSearch?

2.Construct validity and capability measurement: Although the paper provides human agreement studies, to what extent do the rubric-based scores reliably measure the underlying capabilities of deep research agents, particularly factual accuracy, evidence synthesis, conflict resolution, and uncertainty handling? Could the authors provide capability-level analyses or additional validation beyond agreement with human judgments?

3.Robustness and generalizability: How sensitive are the reported model scores and rankings to rubric-design and task-selection choices? In particular, would the conclusions remain consistent when implicit requirements are excluded, alternative rubric weights are used, task-domain compositions are changed, or semantically equivalent task rephrasings are evaluated?

1 comment

Thank you for the detailed response and the additional analysis conducted. The sensitivity checks and additional LLM-judge evaluation do address my concerns about the stability of the reported rankings under the settings described in the paper. However, the distinctive scientific contribution of LifeSearch remains insufficiently established, as it is not demonstrated which of LifeSearch's capabilities existing SOTA benchmarks fail to capture. The summary table of the comparison between LifeSearch and other SOTA models did not convincingly show the additional capability or gap that LifeSearch addresses but other SOTA models do not. The only differences seem to be increasing the number of domains covered to 7 and including bilingual tasks, but these are design choices rather than the inherent limitations of other SOTA models. It is almost certain that other SOTA benchmark models can also cover 7 or perhaps more domains if needed.

My main remaining concern is the interpretability of capability scores. Assigning capability labels to existing rubrics using another LLM provides a useful breakdown of report quality, but does not by itself justify the stronger claim that the LifeSearch agent can isolate the underlying capabilities of a DR agent or explain why a DR agent fails. For example, a failed factual-accuracy criterion could reflect missing retrieval, incorrect synthesis, or an omission from the final report, depending on the DR agent and that specific task instance. The LLM judge might decide on one failure cause, but in another instance decide on another failure cause, as this is the inherent probabilistic and often non-interpretable nature of the LLM decision-making process. It is always possible to have a human expert act as the final judge and manually adjudicate the individual cases, but this defeats the purpose of having a LifeSearch benchmark that should ideally provide some theoretical guarantee on the correctness of its evaluation decision. 

I appreciate the improvements provided in the rebuttals, but maintain my weak-reject recommendation because the contribution and the capability-measurement claims remain only partially supported.

Reviewer_4o1rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LifeSearch is a new bilingual (English and Chinese) deep research benchmark comprising 203 tasks across 7 life and productivity domains. The main contribution of the work is that it has been built from real user queries and graded using 4014 expert-written, weighted rubrics. LLM-as-judge works as the final evaluation. Gemini 3.1 Pro is the language model used for the evaluation setup. Among the 12 agents evaluated, models show poor performance, with the best being GPT-5.6 Sol, reaching 63.4%.4%.

Critical strengths

1- Real user query seeds a large human annotation effort 2- About 40% of rubrics name concrete values, events or documents, which makes them easier to verify than holistic "quality" criteria 3- Judge configurations are compared openly, including self-consistency across runs

Critical weaknesses

1- Reliance on an LLM judge is a problem. Previous research has tried this idea, but they all have seen different limitations with it. In the case of this paper, the downsides are: (1) Judge accuracy is only about 84 against human labels. Many gaps between models are much smaller than that error rate (for instance 49.73 vs 49.19) (2) The judge was validated only on 20 reports, one per task, from a single unnamed model. (3) There is no check on whether the model ranking stays the same with a different judge. The reason behind choosing Gemini is unclear to me. (4) We see Gemini 3.1 Pro grades Gemini 3.6 Flash. That might result in biased judgement. (5) The error analysis in Table 4 only covers variations across runs but not the judgement. (6) Accuracy should be split by rubric type but that has not been reported at all.

2- The analysis of prior work is limited and incomplete. Even mischaracterized a bit. The paper says prior benchmarks focus on "academic research or synthetic tasks", but this overlook DRBench from ICLR 2026 (https://proceedings.iclr.cc/paper_files/paper/2026/hash/0ba520d93c3df592c83a611961314c98-Abstract-Conference.html). DRBench citation and discussion is missing. DRBench targets enterprise deep research which is a real-world setting. Its files are synthetic, but they are grounded in real web information. Key parts including the main insights are also human written and annotated as well as the questions.

3- "Real-user" claim weakened by heavy rewriting. This makes the final tasks partly constructed which is against what paper criticizes about the prior benchmarks.

4- Human study is thin and inconsistent. There is also a mismatch between Table 5 and 4. Table 4 puts Kimi-K3 above Doubao and Claude Opus 4.8 above Hy3; Table 5 reverses both. The subset scores should be reported.

5- Why the agents are run in a shared ReAct harness rather than a native DR product.

6- Rubrics only check what they list. Rubrics are designed so the judge never browses. I know this limitation exists in the prior benchmarks but works like DRBench have the "Distractor Avoidance" metric which helps to quantify it.

Questions for authors

1- How do the model rankings change if you grade with a second judge or with an ensemble of judges? 2- What is judge accuracy per agent and per rubric type? 3- How were the 7 systems for the human study chosen and why don't their ranks match Table 4? 4- Will the tasks, rubrics and cached and web pages released? 5- I believe it worth to analyze the judge models (Jev, Kev, etc.) for the future versions (I am aware they were not introduced at the time of submission, but it will be good to include for the future versions)

Reviewer_RiBkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LifeSearch is a benchmark for deep research on life and work questions. It has 203 Chinese and English tasks and 4,014 scoring rules. The authors start with user requests from an internal product, then filter and rewrite them. Experts write rules about content, facts, and sources. The authors test 12 systems in a shared tool setup, with three runs per task. They choose an LLM judge using human labels and compare benchmark ranks with human ranks. The best system reaches about 63.4% on the benchmark.

Critical strengths

I think the data can be useful. The rules ask for concrete numbers, units, dates, names, and sources. The Chinese and English coverage is also useful.

Critical weaknesses
  • I think the novelty claim is too strong. Table 1 marks earlier benchmarks as lacking life and productivity coverage. But DRACO has real user requests, product comparisons, and personal-assistant tasks. ResearchRubrics includes consumer research and business planning. LiveResearchBench covers daily life, enterprise, and academia. I find the table misleading. The bilingual data and detailed fact targets can still add value, but I am not convinced by the “first benchmark” claim as written.
  • I am unsure whether the final tasks keep the original user needs. An LLM removes file dependencies and operation requests. Experts then make the tasks harder to test certain skills. I am not sure whether the final rewritten tasks still match real use. I think this needs more before-and-after examples and checks with the users. Otherwise, the broad claim about life and work is hard to judge.
  • I am not convinced that the judge has enough independent testing. The 402 labels come from 20 tasks, with one model answer per task. The authors use them to choose among 11 judge settings and report about 84.25% accuracy on the same labels. The human rank study covers seven systems on 30 tasks and leaves out several close competitors. I do not think the 0.89 rank correlation settles the close comparisons.
Questions for authors
  • What exact gap do you claim over DRACO, ResearchRubrics, and LiveResearchBench? Can you revise Table 1 to reflect their life and work tasks?
  • Can you show original and rewritten requests with private details removed? Did the original users check whether the final tasks still match what they wanted?
  • Do you have a held-out judge test on new tasks and outputs from different models? Please show errors by language and model, and uncertainty for close system comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.