- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that in personalized deep research, a user profile should be treated as evidence about the current-task intent rather than as the intent itself, and proposes AskInfer-Bench to evaluate this distinction, which comprises three components: 1) Human-Grounded Preferences: 15 long-form tasks whose reference preferences are provided by task owners, yielding 98 high-impact preference axes and 409 semantic rubric leaves; 2) Controlled Evaluation Protocol: the visible profile is masked at three levels (cold start, RAW50, and RAW100), clarification access is toggled independently, and a frozen selective-disclosure user simulator answers questions from the task owner's latent state; and 3) Human-Grounded Report Scoring: final reports are scored against the human-provided rubric by three blinded model judges. The paper first shows that persona-derived criteria generated by a PDR-Bench-style generator achieve high recall mainly through overgeneration, and that recall drops sharply under a matched criterion budget. Experiments with 360 reports across two deep-research harnesses show that richer profiles improve report scores but saturate, and that clarification with an IEO question selector helps under sparse or partial profiles but hurts under full profiles.
Critical strengths
- The paper raises a valid concern about current personalization benchmarks: rubrics derived from the same persona given to the agent may turn model inferences into ground truth.
- The evaluation protocol separates profile exposure from clarification access, and collects reference preferences from task owners before showing them any prompts.
- The authors clearly discuss the limitations of their study and avoid overclaiming general causal conclusions.
Critical weaknesses
-
The scale is too small to support a benchmark. The benchmark contains only 15 tasks, each grounded in a single participant, and the source and elicitation protocol of five of them are not described. With one owner per task, the reference preferences reflect a single one-time statement, and neither within-person stability nor between-user variation is examined, so it is unclear whether the conclusions reflect general agent behavior or the idiosyncrasies of a few participants.
-
Key validations are incomplete. 1) The user simulator, which determines what clarification can obtain, has not been audited in production: only 35 of 139 interactions were retained, selected by availability rather than at random, 0/35 have been audited, and the development probe used a different configuration; 2) the persona-vs-human rubric comparison relies on two LLM judges with low agreement (κ = 0.23–0.39) and no human adjudication, and its strict recall changes from 80.8% to 41.1% depending on the judge; 3) the appendix still lists an open audit item "before submission" and a "not yet computed" entry, suggesting that the study was not finished at submission.
-
The statistical evidence is weak. The reported clarification effects are about ±0.7 on a 0–10 scale, which is comparable to the judge error (MAE of 4.2 on a 0–100 scale). The bootstrap intervals for ODR are computed over only five task clusters, no significance tests are reported, and the dual-scoring analysis that most directly supports the central claim uses only five report pairs.
-
The main finding lacks mechanistic analysis. The reversal of the clarification effect under full profiles is the most interesting result, but the paper does not analyze which questions are asked, whether they target information already present in the profile, or whether the answers are reflected in the report. It also does not rule out a simpler explanation: under fixed research and token budgets, clarification may consume resources that matter more when the context is already long. Moreover, the effect is obtained with the authors' own IEO selector, while the native DeerFlow policy rarely asks, so it is unclear whether it generalizes to other agents.
-
The contribution and presentation are limited. The observation that model-derived rubrics may diverge from real user preferences has been discussed in prior work [1, 2, 3], and the benchmark offers neither a scalable construction method nor a new clarification approach. The writing is also hard to follow, with many version numbers, hashes, and overly precise values (e.g., "CNY 4.971342", "−0.2508"), which read more like an experiment log than a research paper.
[1] Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users. ACL, 2026.
[2] RubricBench: Aligning Model-Generated Rubrics with Human Standards. arXiv:2603.01562, 2026.
[3] Re-Centering Humans in LLM Personalization. arXiv:2606.06614, 2026.
Questions for authors
see weaknesses