acceptodds
Under review as a conference paper at ICLR 2027

Retrieval Is Not Understanding: The Evidence-to-State Bottleneck in Personalized LLMs

Abstract

Long-term personalized large language models (LLMs) must use information from prior interactions to support decisions in the current context. Memory-centric systems have made substantial progress in storing, retrieving, and organizing historical information, while recent work has also introduced explicit user representations. However, it remains unclear how much user-state error persists when retrieval failure is explicitly removed. We isolate this post-retrieval problem as the Evidence-to-State bottleneck. Under an Oracle Evidence setting, task-relevant historical evidence is directly provided, allowing us to evaluate whether the model can recover the corresponding benchmark-defined user state without confounding retrieval errors. Using PersonaMem-v2, we represent each state by its type, subject, content, and applicability. Even under this favorable setting, a base LLM achieves only 19.81% Full-State Accuracy, indicating substantial residual error in user-state interpretation. To test whether this bottleneck can be substantially reduced with direct supervision, we apply a simple controlled state-supervision intervention, which raises Full-State Accuracy to 65.28%. We then perform matched state replacement while holding the query, Oracle Evidence, candidate options, and downstream decision model fixed. Replacing base-generated states with improved states raises personalized decision accuracy from 47.10% to 68.81%, recovering 78.03% of the decision gap between the base-state and reference-state conditions. Error-level analysis further shows that different state-error categories are associated with markedly different downstream decision gaps. These results show that making the right user evidence available is not sufficient for reliable personalization: models must also correctly interpret what that evidence implies about the user. Our findings motivate treating Evidence-to-State reasoning as an explicit evaluation target in personalized LLM systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, personalization, user modeling, user state, long-term memory, personalized decision making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nej5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the LLM infers the user's state under personal scenarios. The author uses evidence from Personamem-v2 and represents each state by type. The performance of Qwen3-4B reaches 19.81% and is raised to 65.28% by SFT.

Critical strengths
  1. The paper distinguishes the retrieval and interpretation phases, which makes the retrieval error easy to identify.

  2. The ablation study is well-designed and only changes user states in the experiments, which makes comparison clearer.

  3. The paper reports the confidence interval at the user level, which can deal with the relevance with the same personal sample.

Critical weaknesses
  1. The experiment part is weak. The author only uses one dataset and doesn't compare with other memory baselines. I think this paper is below the ICLR bar.

  2. Only a weak 4B model is tested under the zero-shot setting. The experiments can not show the real problem of LLM.

  3. The paper says retrieval is not enough, but no experiment uses retrieval phase. Moreover, the paper also does not report how many errors come from the retrieval stage.

  4. The main motivation is model cannot understant user, but the results cannot support this view.

Questions for authors
  1. Could the authors add experiments on more datasets and compare with more memory baselines?

  2. Could the authors test stronger LLMs and also report their results?

  3. Could the authors add experiments with retrieval?

Reviewer_HuAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Long-term personalized LLMs retrieve information from a user’s history and then use it for the current decision. The paper asks how much error remains after retrieval has succeeded. It decomposes personalization into history, task-relevant evidence, an inferred user state, and a final decision. Using PersonaMem-v2, it isolates the evidence-to-state step by supplying “oracle” evidence directly. It defines a structured, benchmark-derived user state (type, subject, content, applicability) and scores a model’s inferred state against this reference, with an LLM judging the free-form content.

It compares a prompted base model with a model fine-tuned on benchmark-derived reference states. In a matched state-replacement experiment, a fixed decision model receives either no state, the base state, the fine-tuned state, or the reference state. It also analyzes how different error types relate to decision accuracy and audits how hard the oracle evidence is. The authors report that the base model recovers few states correctly even with oracle evidence, that fine-tuning greatly improves this, and that better states improve decisions. They also report that different error types have different downstream costs. They conclude that retrieving the right evidence is not the same as understanding the user.

Critical strengths

The diagnostic design is clean and clearly motivated. Decomposing the pipeline and holding the query, evidence, options and decision model fixed while only the supplied state changes (conditions D0–D3) is a sensible way to ask whether state quality matters downstream. The authors use a paired persona-level bootstrap to account for clustering within personas, report paired outcomes (343, 33, 896 and 626 examples), and break results down by state type. Their limitations section is frank about the single dataset, the benchmark-defined reference states, and the missing held-out test set.

The error-level analysis (Figure 3) and the 200-example evidence audit (Table 8) address natural objections, namely that errors differ in downstream cost and that oracle evidence might make the task trivial extraction. The training configuration is specified, and the headline numbers are internally consistent.

Critical weaknesses

The central measurement may mostly reflect conformance to the benchmark’s label scheme and reference phrasing, not interpretation of the evidence. After fine-tuning on 16,631 benchmark-derived examples, type, subject and applicability accuracy go from 46.73, 68.07 and 61.59% to 98.37, 98.95 and 98.89% (Table 2). In the audit (Table 8), the base model scores 8.33% on examples where the state is directly stated in the evidence (3 of 36), below its 20.22% on attribution/scope and 28.57% on deletion cases. Content-only semantic accuracy is never reported (only exact content match, 6.22% to 34.98%). Conditional on type, subject and applicability all being correct, full-state accuracy is 45.1% for the base model and 66.4% for the fine-tuned one (my calculation). Depending on how the gain is decomposed, 54–80% of the 45.47-point gain therefore comes from the categorical fields (my calculation). The reference states are described as not uniquely identifiable (Section 3), yet a single reference judges open-ended content, using an unnamed LLM validated on 30 examples (Appendix C). The state-inference results use one 4B model (Qwen3-4B) with a single prompt design, so “a base LLM” in the abstract goes beyond what is tested.

The downstream evidence is weaker than the headline suggests. The fine-tuned state is trained to reproduce the reference state (Section 4), so approaching the reference-state condition (D3) is expected. The experiment does not establish that evidence-to-state interpretation is what limits decisions in a realistic pipeline. There is no comparison with a decision model prompted or fine-tuned to reason directly from the evidence, or with a stronger decision model. The decision model is unnamed, the number of answer options is not given, and no non-oracle baseline is reported. The error analysis (Figure 3) covers four categories (N=142, 31, 231 and 253) with no intervals. Decision accuracy under the reference state itself ranges from 58.06% to 91.34% across categories, and states labeled unsupported inference or over-generalized still reach 78.79% and 81.03%, above the overall D2 accuracy of 68.81%. The gaps in Figure 3 therefore cannot be attributed to the error type alone.

The scope is narrower than the title. No retrieval system is evaluated, and the oracle evidence is described only as “the task-relevant evidence supplied by the benchmark” (Section 3), with no account of how it was built. The study uses one benchmark, one development set of 80 personas and no held-out test set (Limitations), and reports no seeds or repeated runs. QUMem, PUMA and MemChain, named in the related work, are not compared. The 200-example audit does not report the number of annotators or their agreement, and the paper does not discuss the base model’s low accuracy on directly stated evidence.

Questions for authors

What is content-only semantic accuracy, independent of type, subject and applicability, overall and per state type? Why does the base model get only 2.97% on PREFERENCE and 8.33% on directly stated evidence (Table 8), and how much of its error is a mismatch with the benchmark’s label conventions?

How were the oracle evidence and the reference states constructed (including the OTHER type and the content field)? Can more than one state be valid for the same evidence, and how does the evaluation treat that?

Which models are the semantic evaluator and the decision model? How many answer options are there (chance level), and how does decision accuracy look with retrieved or full history instead of oracle evidence?

The results use one 4B model and one prompt. How do larger or frontier models, few-shot prompting and reasoning prompts perform, and does the bottleneck persist?

The fine-tuned state is trained to reproduce the reference state. What shows that interpretation, not imitation of benchmark labels, drives the decision gains, and how does the pipeline compare with direct decision prompting or fine-tuning with the same evidence?

All analyses use one development set with no held-out test set. Were prompts, checkpoints or epochs chosen on it, and how many runs were done? For the audit, how many annotators were there, how well did they agree, and did they see the reference state?

Reviewer_bce4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper decomposes personalized LLM decision-making into history → evidence → user state → decision, and isolates the evidence-to-state step by providing Oracle Evidence. On PersonaMem-v2, Qwen3-4B reaches only 19.81% Full-State Accuracy even with the correct evidence, and standard SFT raises this to 65.28%. In a matched state-replacement experiment, where the query, evidence, options, and decision model are held fixed, swapping base states for SFT states raises decision accuracy from 47.10% to 68.81%. The authors argue that retrieving the right evidence is not the same as understanding the user, so evidence-to-state reasoning should be evaluated as a separate target.

Critical strengths

1.Clean experimental control. Oracle Evidence removes retrieval error, and the state-replacement design varies only the state source, which makes the comparison easy to interpret. 2.Sound statistics. Persona-cluster bootstrap, paired confidence intervals, and breakdowns by state type, error type, and evidence difficulty. 3.Honest positioning. SFT is presented as a controlled intervention rather than a new method, and the limitations are stated clearly.

Critical weaknesses

1.Limited novelty. The work amounts to an ablation study. Oracle upper-bound analysis, mediation-style state replacement, and error breakdowns are all standard tools. The paper proposes no new method and offers no explanation of why the bottleneck exists. 2.The bottleneck may be overstated. The baseline omits few-shot and CoT prompting. Most of the SFT gain appears in the categorical fields (state type accuracy goes from 46.73% to 98.37%), which suggests the model is learning the benchmark's label conventions rather than better user understanding. Training targets and evaluation targets come from the same benchmark metadata, which makes this explanation more likely. 3.The narrative is only partly supported. Even with reference states, decision accuracy reaches just 74.92%, so a substantial share of the error sits in the decision stage. This complicates the claim that evidence-to-state is the key bottleneck, and the paper does not discuss it. 4.Narrow validation. One dataset, one 4B model, and no held-out test set.

Questions for authors

Same as weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.