acceptodds
Withdrawn from ICLR 2027

AskInfer-Bench: Profiles Are Evidence, Not Intent Labels, in Personalized Deep Research

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized deep-research agents are often evaluated using criteria derived from the same persona supplied to the agent. This tests responsiveness, but it can silently turn a model's inference about a user into ground truth about the user's current task. We frame personalization under incomplete context as an information-acquisition problem: a profile is evidence about task intent, not the intent itself. AskInfer-Bench contains 15 tasks whose reference preferences are all human-provided, varies how much persistent profile evidence is initially visible, and independently controls clarification access. In the audited Prolific batch, ten adults supplied 107 raw statements, consolidated into 73 high-impact task-preference axes and 22 secondary preferences. A PDR-Bench-style GPT-5 generator produced 379 criteria for those ten tasks. Although its full lists attained 80.8% strict recall under an LLM-favorable semantic merge, recall fell to 39.7% after matching the 73-criterion human budget using the generator's own weights. The full-factorial system study contains 360 reports across two harnesses. Native clarification has near-zero average effects in the complete Gemini–DeerFlow matrix, whereas Open Deep Research with an Influence–Evidence–Ownership (IEO) question selector yields positive Ask–Off effects for three backbones under cold and RAW50 context but negative effects under RAW100. A separate five-pair dual-scoring diagnostic changes the average Ask effect from -0.2508 under human-grounded rubrics to +0.2924 under persona-derived rubrics and reverses the win/tie/loss conclusion for two pairs. This small diagnostic is not a population estimate or significance test; it illustrates why personalization benchmarks must separate using supplied information from deciding what still needs to be learned.

Primary area:
datasets and benchmarks
Keywords:
deep research agents, personalization, clarification, intent elicitation, user modeling, human-AI interaction, agent evaluation, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Xgc6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that in personalized deep research, a user profile should be treated as evidence about the current-task intent rather than as the intent itself, and proposes AskInfer-Bench to evaluate this distinction, which comprises three components: 1) Human-Grounded Preferences: 15 long-form tasks whose reference preferences are provided by task owners, yielding 98 high-impact preference axes and 409 semantic rubric leaves; 2) Controlled Evaluation Protocol: the visible profile is masked at three levels (cold start, RAW50, and RAW100), clarification access is toggled independently, and a frozen selective-disclosure user simulator answers questions from the task owner's latent state; and 3) Human-Grounded Report Scoring: final reports are scored against the human-provided rubric by three blinded model judges. The paper first shows that persona-derived criteria generated by a PDR-Bench-style generator achieve high recall mainly through overgeneration, and that recall drops sharply under a matched criterion budget. Experiments with 360 reports across two deep-research harnesses show that richer profiles improve report scores but saturate, and that clarification with an IEO question selector helps under sparse or partial profiles but hurts under full profiles.

Critical strengths
  1. The paper raises a valid concern about current personalization benchmarks: rubrics derived from the same persona given to the agent may turn model inferences into ground truth.
  2. The evaluation protocol separates profile exposure from clarification access, and collects reference preferences from task owners before showing them any prompts.
  3. The authors clearly discuss the limitations of their study and avoid overclaiming general causal conclusions.
Critical weaknesses
  1. The scale is too small to support a benchmark. The benchmark contains only 15 tasks, each grounded in a single participant, and the source and elicitation protocol of five of them are not described. With one owner per task, the reference preferences reflect a single one-time statement, and neither within-person stability nor between-user variation is examined, so it is unclear whether the conclusions reflect general agent behavior or the idiosyncrasies of a few participants.

  2. Key validations are incomplete. 1) The user simulator, which determines what clarification can obtain, has not been audited in production: only 35 of 139 interactions were retained, selected by availability rather than at random, 0/35 have been audited, and the development probe used a different configuration; 2) the persona-vs-human rubric comparison relies on two LLM judges with low agreement (κ = 0.23–0.39) and no human adjudication, and its strict recall changes from 80.8% to 41.1% depending on the judge; 3) the appendix still lists an open audit item "before submission" and a "not yet computed" entry, suggesting that the study was not finished at submission.

  3. The statistical evidence is weak. The reported clarification effects are about ±0.7 on a 0–10 scale, which is comparable to the judge error (MAE of 4.2 on a 0–100 scale). The bootstrap intervals for ODR are computed over only five task clusters, no significance tests are reported, and the dual-scoring analysis that most directly supports the central claim uses only five report pairs.

  4. The main finding lacks mechanistic analysis. The reversal of the clarification effect under full profiles is the most interesting result, but the paper does not analyze which questions are asked, whether they target information already present in the profile, or whether the answers are reflected in the report. It also does not rule out a simpler explanation: under fixed research and token budgets, clarification may consume resources that matter more when the context is already long. Moreover, the effect is obtained with the authors' own IEO selector, while the native DeerFlow policy rarely asks, so it is unclear whether it generalizes to other agents.

  5. The contribution and presentation are limited. The observation that model-derived rubrics may diverge from real user preferences has been discussed in prior work [1, 2, 3], and the benchmark offers neither a scalable construction method nor a new clarification approach. The writing is also hard to follow, with many version numbers, hashes, and overly precise values (e.g., "CNY 4.971342", "−0.2508"), which read more like an experiment log than a research paper.

[1] Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users. ACL, 2026.

[2] RubricBench: Aligning Model-Generated Rubrics with Human Standards. arXiv:2603.01562, 2026.

[3] Re-Centering Humans in LLM Personalization. arXiv:2606.06614, 2026.

Questions for authors

see weaknesses

Reviewer_3fGrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AskInfer-Bench includes 15 tasks to address a flaw in a previous personalized deep research agent: it treated user profiles as labels, not task intent. Profile exposure and clarification access are varied independently. Two findings are reported. First, PDR-Bench-style persona-derived rubrics reach high recall mainly by overgenerating criteria. Second, clarification helps with cold or partial context but hurts once the full profile is visible.

Critical strengths

1- The conceptual point is valid and useful. Using the same persona to build both the agent's input and the rubric turns the model's inference into ground truth. 2- Varying profile exposure and clarification access as separate factors is a clean design. 3- The automated report judge is validated against human labels.

Critical weaknesses

1- This is clearly an unfinished work. Human audits, simulator checks, and RQ1 matching remain incomplete. 2- Only 15 participants/tasks, five undocumented, with effects below 1/10 point. 3- Missing intervals, an unexplained five-task bootstrap, and acknowledged low precision. 4- Unreliable RQ1 evaluation. Judges disagree substantially the headline favors one judge and rubric granularity may explain the gap. 5- The "misdirected questions under rich context" explanation is never checked, and simpler explanations go untested: "undecided" simulator answers, the token budget (7k/6k/9k), or the four-question cap. 6- Gemini/DeerFlow asks in only 9 of 45 episodes, so its near-zero effect says little. The GPT/DeerFlow arm is incomplete (8–9 tasks) but still appears in the table and figure 7- The paper is dense with version numbers, internal field names and audit jargon, and the results are given without interpretation. The abstract is mostly numbers.

There are many TODOs remaining from the LLM writing: Appendix I, Appendix K.1, Appendix K.2. More problems: (1) False precision and jargon: costs reported as "CNY 4.971342", effects as "-0.2508", plus internal labels like "frozen v0.107 simulator", "v0.84 P", and "historical fixed-eight-unit acquisition metric". (2) Markdown artifacts: the citations in Related Work render a literal "(" (e.g., "(Liang et al., 2026)"), and the Table 1 caption has broken quotation marks. (3) Repetitive defensive hedging stands in for analysis. "Not a population estimate", "descriptive, not causal" and "an evaluation scaffold, not a claimed algorithmic contribution" each appear several times. (4) The AI use statement admits AI was used for "drafting… interpretation and presentation of empirical results".

Questions for authors

1- Was the study approved by an ethics board, and what consent form did participants actually see? 2- Which tasks are in the ODR bootstrap, and what are the intervals for Table 1? 3- What explains the negative Ask effect at RAW100? How often does the simulator answer "undecided", and does it differ by context level?

Reviewer_MeEMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AI Statement: I don't use AI for generating reviews - that is a abdication of duty, and I do not think LLM generation has a place in academic discourse - LLM generation in papers is plagiarism and poor scholarly practice.

Critical strengths

The paper develops another benchmark AskInfer-Bench and used human participant data to perform tests.

Critical weaknesses

The paper is very unclear. It appears to make sense at the surface, but is full of buzz words, adjectives and descriptors which appear to convey meaning but actually don't. It is full of undefined terms. The level of readability of this paper is, I am afraid, too low for me to really understand what is going on in the paper. As such I can only defer this until it is better prepared.

By way of example. Even in the abstract: "Personalized deep-research agents are often evaluated using criteria derived from the same persona supplied to the agent. This tests responsiveness, but it can silently turn a model’s inference about a user into ground truth about the user’s cur- rent task. We frame personalization under incomplete context as an information- acquisition problem: a profile is evidence about task intent, not the intent itself. AskInfer-Bench contains 15 tasks whose reference preferences are all human- provided, varies how much persistent profile evidence is initially visible, and in- dependently controls clarification access. In the audited Prolific batch, ten adults supplied 107 raw statements, consolidated into 73 high-impact task-preference axes and 22 secondary preferences. A PDR-Bench-style GPT-5 generator pro- duced 379 criteria for those ten tasks. Although its full lists attained 80.8% strict recall under an LLM-favorable semantic merge, recall fell to 39.7% af- ter matching the 73-criterion human budget using the generator’s own weights. The full-factorial system study contains 360 reports across two harnesses. Native clarification has near-zero average effects in the complete Gemini–DeerFlow ma- trix, whereas Open Deep Research with an Influence–Evidence–Ownership (IEO) question selector yields positive Ask–Off effects for three backbones under cold and RAW50 context (+0.71 to +0.78 and +0.27 to +0.34) but negative effects under RAW100 (-0.63 to -0.74). A separate five-pair dual-scoring diagnostic changes the average Ask effect from -0.2508 under human-grounded rubrics to +0.2924 un- der persona-derived rubrics and reverses the win/tie/loss conclusion for two pairs. This small diagnostic is not a population estimate or significance test; it illustrates why personalization benchmarks must separate using supplied information from deciding what still needs to be learned"

I have no idea what "tests responsiveness" means (to what?), what "silently turn a model’s inference about a user into ground truth about the user’s current task." means (silent in what way, what inference is a model trying to do and what is the ground truth about the current users task - how can you turn a model inference into a ground truth, by definition of "ground truth" that is not possible) etc. This goes on throughout the paper. The purpose of the paper is ill-defined in my reading. As a result this falls at first hurdle to me. I am not really able to evaluate the paper as a whole as it is not written or constructed as a scholarly work, establishing the basis of it premise and building reliable conclusions upon that.

Questions for authors

I have no questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.