- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PairPref tests whether assistants use a persistent user preference when appropriate and set it aside when circumstances change. It contains 1,227 pairs covering 45 preferences and eight situation categories. Each pair shares the preference, request, and candidate replies but differs in context. The authors evaluate eight models through response selection and free generation, finding frequent preference use in both contexts even when models select the intended answers.
Critical strengths
- The paired design provides a useful test of contextual preference use, requiring models to both follow and set aside the same preference as circumstances change.
- The comparison between selection and generation is informative, particularly the finding that models often retain the preference in both generated answers, even on pairs where they select the intended replies.
- The paper is well organized and easy to follow.
Critical weaknesses
- Human validation provides limited support for the evaluated dataset, as it uses revised situations without detailing the changes or their impact on model evaluation (§3.3).
- The automatic generation judge has not been validated against human ratings, leaving the reliability of its preference-use and situational-fit scores unclear.
- Gen-OK may penalize reasonable answers because it requires zero preference use in negative contexts, even when situational fit is fully satisfied (§4.1). Whether this excludes acceptable preference accommodation remains unclear.
- The lack of results by situation category and the positive-context clean/distractor condition makes it difficult to identify where failures concentrate.
- The effect of context-to-request distance is not analyzed, despite the dataset including distances of one, three, and six rounds.
- Removing all memories does not isolate the target preference’s effect from that of companion memories (Table 3). Removing only the target preference would clarify its contribution to preference use and situational fit.
- The experiments lack statistical significance testing.
Questions for authors
- Could you clarify how the human validation supports the unrevised dataset used for model evaluation?
- How well do the automatic generation scores agree with human judgments?
- How does the evaluation distinguish inappropriate preference use from reasonable accommodation that satisfies the negative context?
- Can results by situation category, clean/distractor condition in positive contexts, and context-to-request distance help identify where models fail and why?
- How do preference use and situational fit change when only the target preference is removed?
We thank all the reviewers for their very helpful feedback. We have decided to withdraw the paper and continue refining it.