acceptodds
Withdrawn from ICLR 2027

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory-augmented assistants use retrieved preferences to guide their responses. A small change in the situation can change whether a preference is appropriate while barely affecting its retrieval similarity. Memory benchmarks typically test whether systems store and retrieve preferences, with less attention to when those preferences should apply. We introduce PairPref, a benchmark of contextual preference use. Each pair changes only the situation, keeping the preference, request, and four candidate replies fixed. The preference remains valid in both situations. In the selection track, models must choose the reply that applies the preference only where appropriate. In the free-generation track, they must decide when to apply it without seeing candidate replies. Both tracks use the same 1,227 pairs across 45 preferences and eight situation categories. We evaluate eight models, most of which achieve selection scores () of 51 to 65 points. In free generation, however, both responses are appropriate for their respective situations in only 3.6% to 18.3% of pairs retained after hedging exclusions. Models continue to apply the preference in both situations even with fewer retrieved memories, alternative presentation formats, and a stricter prompt. These results show that models still struggle to judge when user preferences apply and respond accordingly.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, contextual applicability, persistent memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TjonOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PairPref tests whether assistants use a persistent user preference when appropriate and set it aside when circumstances change. It contains 1,227 pairs covering 45 preferences and eight situation categories. Each pair shares the preference, request, and candidate replies but differs in context. The authors evaluate eight models through response selection and free generation, finding frequent preference use in both contexts even when models select the intended answers.

Critical strengths
  1. The paired design provides a useful test of contextual preference use, requiring models to both follow and set aside the same preference as circumstances change.
  2. The comparison between selection and generation is informative, particularly the finding that models often retain the preference in both generated answers, even on pairs where they select the intended replies.
  3. The paper is well organized and easy to follow.
Critical weaknesses
  1. Human validation provides limited support for the evaluated dataset, as it uses revised situations without detailing the changes or their impact on model evaluation (§3.3).
  2. The automatic generation judge has not been validated against human ratings, leaving the reliability of its preference-use and situational-fit scores unclear.
  3. Gen-OK may penalize reasonable answers because it requires zero preference use in negative contexts, even when situational fit is fully satisfied (§4.1). Whether this excludes acceptable preference accommodation remains unclear.
  4. The lack of results by situation category and the positive-context clean/distractor condition makes it difficult to identify where failures concentrate.
  5. The effect of context-to-request distance is not analyzed, despite the dataset including distances of one, three, and six rounds.
  6. Removing all memories does not isolate the target preference’s effect from that of companion memories (Table 3). Removing only the target preference would clarify its contribution to preference use and situational fit.
  7. The experiments lack statistical significance testing.
Questions for authors
  1. Could you clarify how the human validation supports the unrevised dataset used for model evaluation?
  2. How well do the automatic generation scores agree with human judgments?
  3. How does the evaluation distinguish inappropriate preference use from reasonable accommodation that satisfies the negative context?
  4. Can results by situation category, clean/distractor condition in positive contexts, and context-to-request distance help identify where models fail and why?
  5. How do preference use and situational fit change when only the target preference is removed?
Reviewer_Wz8HOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on evaluating when stored preferences should guide an answer. In the proposed contextual preference task, each pair of inputs share the same stored preference, request wording, and candidate replies, but the task circumstances are varied. They find that models struggle to apply preference if and only if it’s contextually appropriate.

Critical strengths

The proposed benchmark provides a highly controlled setting to evaluate contextual preferences. This controlled design enables more targeted evaluation on model capabilities, but more rigorous validation is needed to draw reliable conclusions.

Critical weaknesses

It is unclear what is the novelty of the paper compared to prior work that studies contextual cues making preference unusable (https://arxiv.org/abs/2503.07018).

The benchmark construction pipeline it vaguely documented. Where does the 45 preferences come from? How does gpt-5.5 generate the request, situations, and how was memory generated? The validation pipeline seems inadequate, where another lexical matching and another LLM evaluates change in applicability, but the Deepseek-v4-flash validation pass itself is not validated. It’s unclear what context-free choice screening is selected for, unclear what model version and size is “Qwen” (L206), and unclear what “situation revisions” (L209) are. The annotator agreement for applicability is suspiciously high, but this is because there is no control in the annotation, if all annotators blindly select “applicable” for all 200 samples, it requires the least amount of effort and leads to 100 agreement. Actual annotations should have 50/50 inapplicable and applicable samples.

The LLM judge used to judge the free-response is also not validated.

The manuscript, from study design to experimentation to the writeup, can benefit from more human efforts.

Questions for authors

“Selection scores” is mentioned twice in the abstract and intro but never defined, but L270-277 define a set of metrics not reported.

Reviewer_JxqVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors have proposed a new benchmark dataset, i.e., PairPref, to evaluate when the language models should use the user memory to generate responses. PairPref includes two tracks of data, i.e., the selection track and the free-generation track. Both tracks use the same 1227 pairs across 45 preferences and 8 situation categories. The authors have performed extensive experiments on PairPref via evaluating 8 models. The experimental results indicate that most of the models achieve selection scores between 51 and 65. For the free generation task, both responses are appropriate for their situtations in only 3.6% to 18.3%.

Critical strengths
  1. The authors introduce a new dataset, named PairPref, to study when LLMs should use the user memory for generating responses. PairPref effectively isolates contextual preference applicability from memory retrieval and preference validity.

  2. The authors have performed experiments to analyze model performances in both response selection and free generation tracks. The experimental results also show insightful findings. The models can often identify appropriate responses, but overuse memory when generating responses.

  3. The experimental evaluation is comprehensive, covering 8 different models, and studying different memory pool sizes, memory presentation, etc.

Critical weaknesses
  1. The novelty of the proposed benchmark dataset seems limited. There are several existing datasets that have studied similar research problems, e.g., RPEval and BenchPreS.

  2. For the free generation task, the success rates are between 3.6% and 18.3%. These experimental results are evaluated using LLMs. There is no human validation to make the results more convincing.

  3. The proposed benchmark dataset studies when LLMs should use user memory for generating responses. However, this paper does not introduce any insights about how to solve this problem.

Questions for authors
  1. What is the novelty of the proposed PairPref dataset, compared to previous datasets RPEval and BenchPreS?

  2. Based on the analysis in this paper, how can we enable LLMs to appropriately use the user memory for generating responses?

  3. Whether the success rates of the free generation task are convincing, if we only use LLM for evaluating?

Comments on the paper

1 comment

We thank all the reviewers for their very helpful feedback. We have decided to withdraw the paper and continue refining it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.