acceptodds
Under review as a conference paper at ICLR 2027

Detecting Preference Change Is Not Enough: Rethinking Reward Model Personalization

Abstract

Personalized Reward Models (PRMs) train on users' past choices to predict future preferences. Temporal personalization often assumes that once preference change is detected, the model should adapt its use of that history. We argue that this collapses three distinct questions: *is there enough evidence to adapt this particular user? If so, how should memory change? And is the resulting adaptation genuinely driven by that user rather than by population-level regularities?* We study these questions prospectively, using only information available before evaluation. We fix a pairwise PRM with one component learned across training users and vary only how a second, user-memory component constructs the specific user's history. We compare four memory modes: *retain* keeps all past choices, *discount* progressively downweights older choices, *reset* rebuilds recent memory after a detected change, and a multiscale mode learns a mixture of causal memories operating at different timescales. We evaluate these modes on controlled synthetic histories, PENS, PRISM, and Chatbot Arena, while testing whether temporal structure is real, individually actionable, and prospectively useful. We uncover an *information paradox*: preference change can be clearly measurable across users, while an individual user's earlier history does not reliably identify, before seeing the future, which evaluated memory rule will work best. Choosing the best rule *after seeing the outcomes* appears to gain 4 to 6 percentage points, but about half of that is selection optimism, and the gain disappears when the rule must be chosen earlier. We then uncover an *intervention paradox*. A detected user-specific change does not determine the right memory response. Under abrupt change and identical alarms, event-triggered reset beats a temporary fast discount by about 1 percentage point, but which response wins depends on how late the alarm arrives and how far back reset rebuilds memory. Finally, exploratory analysis reveals an *illusion of personalization*. We find that the PRM can appear to adapt to a particular user while partly acting on population-level regularities rather than reliable evidence about that user's temporal state, helping some users and harming others without reliably identifying who has actually changed. Even the same held-out stable users receive much shorter assigned memory when the PRM is trained in a population where changes are common. Removing history length from the controller greatly weakens this behavior. Our findings show that *detecting change is not knowing whom to adapt, knowing whom to adapt is not knowing how to adapt, and adapting differently is not proof of genuine personalization*.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personalized reward models, preference drift, temporal adaptation, memory operations, prequential evaluation, change-point detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_i6HpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether detecting preference change is sufficient to justify memory adaptation in personalized reward models. Using a shared Bradley–Terry predictor, it compares retention, discounting, multiscale memory, and event-triggered reset. Experiments on synthetic and natural histories distinguish population-level temporal signal from individual actionability, change detection from intervention choice, and adaptive memory allocation from evidence of individual preference change.

Critical strengths
  1. The paper addresses a meaningful evaluation problem and clearly distinguishes detecting preference change from deciding whether and how memory should adapt.

  2. The prospective evaluation, selection-optimism decomposition, and shared-alarm comparisons provide useful controls against misleading conclusions about adaptation.

  3. The experiments cover several temporal regimes and natural datasets. The authors transparently report negative findings and distinguish prespecified results from exploratory analyses.

Critical weaknesses
  1. The information paradox mixes limited individual evidence with changes in the best memory rule across windows. Eeven knowing the true best earlier-window expert can hurt later performance, while longer histories yield positive selection gains. These results warrant a clearer separation between estimation difficulty and temporal transfer failure.
  2. The strongest findings rely on synthetic histories and a linear predictor over frozen representations. Natural PRISM and Arena results are mostly inconclusive, PENS uses a constructed click surrogate, and Arena splices represent switches between users. The checkpoint analyses do not repeat the full selection or attribution tests, limiting validation of the central claims in realistic PRMs.
  3. The conceptual contribution beyond established selection bias, forgetting, and change-detection principles needs sharper justification.
Questions for authors
  1. Can the prospective-selection analysis separately quantify finite-sample estimation error and changes in the true best expert between windows? Would the conclusion hold when both windows lie within the same stable post-change regime?
  2. Does the population-dependent shortening persist when memory is measured using operative lag contributions or prediction changes after removing older observations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer i6Hp: Resolving Technical Concerns and Clarifying Scientific Novelty

We thank the reviewer for the careful comments. We conduct two tests that separate why earlier-window rule selection can fail and show that shorter memory for stable users changes actual predictions. We respond in order of the implications of the concerns.


Q1 [W1; Reviewer Q1]: Does selection fail because of noise or because the best rule changes?

Both can matter. Using the known choice probabilities in our controlled data, we divide the hindsight–prospective gap into 3 parts (Fig. 4 / Table 29.):

Source of gap (points)StableMainChange-point
Hindsight optimism1.862.992.45
Cost of selecting from observed earlier choices1.210.930.90
Cost of using the earlier best rule later0.401.742.51
Total3.475.665.87

Even when preferences never change, the gap is 3.47 points. Different candidate comparisons and nearly tied rules can change which rule is best. The truly best rule differs between windows for 15% of stable users and 42% in the change-point mixture. The cost of using the earlier best rule rises from 0.40 to 2.51 points. This helps explain why the earlier-window oracle loses 0.87 points against the common rule in Table 14.

The stable-only experiment confirms that the problem exists even without preference change. The three-part analysis then shows how much comes from limited evidence and how much from the best rule changing across windows. Our longer-history experiments also show that more observations can improve selection.

Key Takeaway. We now separate and measure the reasons behind the Information Paradox. Limited evidence can make rule selection unreliable even for stable users, while changes in the best rule create an additional challenge.


Q2 [Reviewer Q2]: Does shorter memory actually affect predictions?

Yes. For the same stable users, the operative mean lag is 29.5 steps with a stable-trained controller and 18.4 with a change-point-trained controller. Nominal memory shows the same pattern (29.2 vs. 15.7).

We also removed older choices. Keeping trained controller weights fixed, we recalculated its memory and inputs from the shorter history. Removing choices older than 16 steps changes the predicted probability of the observed choice by 12.3 points for the stable-trained controller versus 7.8 for the change-trained controller. The paired difference is 4.57 points [3.92, 5.21] across five seeds. The same ordering holds at 8- and 32-step cutoffs and without the history-length input (Fig. 9; Appendix P.2).

Key Takeaway. The shortening changes predictions. The change-trained controller relies less on older choices, even for stable users.


Q3 [W2]: Why use controlled histories and a fixed predictor rather than an existing personalized reward model?

We do not claim existing PRMs fail. We keep the predictor and representations fixed across memory rules so differences reflect how history is used. Controlled histories tell us whether and when preferences truly changed, while hiding that information from the tested policies. Note that natural user logs do not provide such ground truth.

We also repeated the selection test using Skywork-Reward-V2-Llama-3.1-8B features on real Arena and PRISM histories. Against common rules chosen from separate training users, earlier-window selection gains -0.75 points on Arena and +0.04 points on PRISM. Neither establishes an improvement (Appendix P.3 and Table 30). This experiment uses features extracted from an existing reward model inside our fixed predictor. It does not test a separately trained personalized reward model with its own learned user representation.

Key Takeaway. Keeping the predictor fixed lets us isolate the effect of memory. Testing a separately trained personalized reward model would tell us whether the findings also hold in another architecture. It does not change what our controlled experiments establish.


Q4 [W3]: What scientific insight is new?

A temporal PRM still faces a chain of decisions: Does this user's history justify adapting? If so, should we retain, discount, or reset memory? Does the resulting memory reflect this user's evidence or is it a population pattern? What we see is that evidence for one decision does not answer the next. This insight does not follow from established results in selection bias, forgetting, and change detection. Our intervention test shows why. Recent evidence beats old evidence, yet keeping old choices helps by 2.33 points. Under the same alarms, reset's advantage over fast discount shifts from -0.28 at zero delay to +1.04 points at eight steps.

Key Takeaway. A change signal does not identify whom to adapt. Knowing whom does not determine how. And a changed memory does not imply individual change. Additional experiments and explanations in the revised manuscript are marked in blue.

Thanks for your efforts. I maintain my positive rating.

Reviewer_8PBbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates temporal personalization in Personalized Reward Models (PRMs), arguing that detecting preference changes alone is insufficient to determine appropriate memory adaptation. The authors formulate three questions: whether available history provides sufficient evidence for individual adaptation, how memory should respond to detected changes, and whether adaptive memory behavior genuinely reflects individual preference changes. To investigate these questions, they adopt a Bradley–Terry preference predictor with a fixed population component and compare several user-memory policies, including retain, discount, event-triggered reset, and the proposed multiscale LETHE approach. Through experiments on controlled synthetic preference histories and real-world datasets (PRISM, Chatbot Arena, and PENS), the paper identifies three phenomena termed the Information Paradox, Intervention Paradox, and Illusion of Personalization. The results highlight the distinction between detecting temporal preference changes, selecting appropriate memory interventions, and interpreting user-specific adaptation.

Critical strengths
  • The paper provides a reasonable framework for understanding temporal personalization by separating preference change detection, memory intervention, and the interpretation of personalized adaptation into three distinct questions. This formulation helps clarify the limitations of treating detected preference changes as sufficient evidence for memory adaptation.
  • The experimental protocol is carefully designed, particularly in its use of prospective evaluation, user-disjoint splits, outcome-disjoint analyses, and controlled synthetic settings. The authors also provide statistical analyses and explicitly distinguish prespecified comparisons from exploratory findings. These efforts make the empirical observations more convincing and reduce potential confounding factors.
Critical weaknesses
  • The findings from synthetic experiments are not consistently reflected in the natural histories. PRISM shows little separation among memory rules, the comparisons on causal Arena have intervals that include zero, and on PENS, LETHE improves over retain (+1.71 pp) but underperforms the learned scalar discount (−0.23 pp) (Section 5, Table 2). It is therefore unclear how far the conclusions from the controlled settings generalize to real-world personalization.

  • The three paradoxes appear largely expected from the problem formulation. The Information Paradox mainly reflects the difficulty of selecting a memory rule from limited and noisy observations, and the Intervention Paradox follows from the fact that a detected change does not necessarily make older evidence harmful. Although the authors position their contribution as the controlled separation of these questions, it is unclear what new insights the findings provide beyond established understanding of model selection and nonstationary learning.

Questions for authors
  • (Clarification) Given the relatively weak or inconsistent separation among memory policies on natural histories, how do the authors interpret the practical implications of the three paradoxes for real-world personalized reward modeling? Are there particular characteristics of real-world histories that explain the differences from the synthetic results?

  • (Clarification) Beyond providing controlled empirical evidence, what do the authors consider the main conceptual novelty of the three paradoxes? In particular, how do these findings extend existing understanding of noisy model selection, preference drift, and memory adaptation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8PBb: Clarifying Conceptual Novelty and Evidence from Natural Histories

We thank the reviewer for recognizing the value of our questions and evaluation. We added analyses that clarify why earlier-window selection fails and whether shorter memory changes predictions. We order our responses by the implications of the concerns.


Q1 [W2; Reviewer Q2]: What is new beyond noisy selection and preference drift?

We agree that selection bias, forgetting, and change detection have known results. To contrast, our contribution is to test three inferences that do not follow from those principles alone: whom to adapt, how to adapt, and whether adaptation reflects change in that individual.

Whom to adapt. In the change-point-heavy mixture, the gap between selecting a rule after seeing outcomes and choosing it earlier has three measured parts: 2.45 points of hindsight optimism, 0.90 from choosing with limited earlier evidence, and 2.51 from using an earlier best rule that no longer works best. Even with unchanged preferences, the gap is 3.47 points. The truly best rule differs between windows for 15% of stable users and 42% of change-point users. We use the generator's expected accuracy to measure the three parts separately, with the same cross-fitted reference. Limited evidence and the failure of a genuinely good earlier rule to transfer are different problems.

How to adapt. Given the same change alarms, reset beats temporary fast discount by 1.05 points in the change-point-heavy mixture. With oracle alarms, its relative result shifts from −0.28 at zero delay to +1.04 points at eight steps. Detection does not determine the response. Nor does more useful recent evidence necessarily mean that older evidence is harmful: in our main mixture, adding the old evidence still improves prediction by 2.33 points over recent-only evidence.

Whether adaptation is personal. For the same stable users, the operative mean lag is 29.5 steps with a stable-trained controller versus 18.4 with a change-heavy-trained controller. Removing choices older than 16 steps changes the predicted probability of the observed choice by 12.3 and 7.8 points on average, respectively. The paired difference in prediction sensitivity is 4.57 points [3.92, 5.21] across five seeds. Shorter memory is reflected in predictions, even for users who never changed. Fig. 4 / Table 29 report the selection decomposition; the operative-memory results are reported in Fig. 9 / Appendix P.2.

Key Takeaway. We test the separate evidence needed to justify a user-specific memory decision, its intervention, and its interpretation.


Q2 [W1; Reviewer Q1]: Why are natural-user-data results less clear?

Controlled histories tell us whether and when preferences truly changed, while hiding that information from the tested models. In contrast, natural histories only show observed choices. Different prompts, candidates, or noise can make preferences appear to change.

The natural results are still useful. PRISM and Arena have short histories, and neither establishes a clear winner among memory rules. PENS has longer histories. Here, LETHE beats retaining all choices by 1.71 points. However, a single learned decay rate beats LETHE by 0.23 points and reset by 0.34 points. This illustrates our central point: even when downweighting older choices helps, it does not tell us which memory update is best.

We also repeated earlier-window selection using features from an 8B Skywork reward model. Against training-selected common rules, gains were −0.75 points on Arena and +0.04 on PRISM (Appendix P.3). Neither establishes an improvement. This is a check using a different reward representation, not a native PRM replication. Note that these results do not establish that PRISM or Arena users were stable, or that PENS users truly changed.

Key Takeaway. Controlled histories help us understand why memory decisions succeed or fail. Natural histories show which decisions actually improve prediction. Both support our central finding: useful temporal evidence does not automatically justify a particular memory response.


Q3 [Reviewer Q1]: What should practitioners learn?

The paper proposes tests for practitioners. It does not imply an automatic memory-reset policy. If recent choices predict better than old ones, a practitioner should test whether removing old choices actually helps compared with using the full history. If a detector fires, one should compare memory responses using the same alarms. If a model shortens memory, one should check whether that behavior is supported by evidence from the particular user.

Key Takeaway. Evidence of change, evidence for a memory action, and evidence of genuine individual adaptation are different. A system should not treat one as proof of the others. We do not claim to have a universal rule that makes these choices automatically. Additional results and explanations in revised manuscript are marked in blue.

Reviewer_8PBbOpenReview ↗

Response to Rebuttal

I thank the authors for the detailed response. The decomposition of the selection gap and the operative-memory analysis address my concern about conceptual novelty to a large extent. For the natural histories, the explanation is reasonable, although the evidence for the three paradoxes still comes mainly from the controlled histories.

I maintain my positive assessment and my rating.

Reviewer_1VW4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper separates detecting preference change, choosing a memory update, and interpreting adaptive user states. It compares retention, discounting, reset, and LETHE within a common pairwise predictor, using synthetic histories and three natural datasets.

Critical strengths

The distinction between recent evidence being better and old evidence being harmful is useful. Predict-before-update evaluation and comparisons under identical alarms make the memory-policy analysis more informative than a simple accuracy ranking.

Critical weaknesses
  1. The connection to personalized reward models remains limited. The strongest findings come from synthetic histories and a fixed Bradley–Terry predictor. The Skywork experiments change the reward signal but do not replicate native personalized models, while PRISM and Arena provide little separation. I would need stronger evidence that the main diagnostics reveal problems in an established personalization system.
  2. The prospective-selection result is difficult to interpret as a practical failure of personalization. Its reference is the best constant expert selected on the evaluation window itself (Appendix F.1). Moreover, Table 14 shows that even an oracle selecting the true best earlier-window expert loses 0.87 points in the change-point setting. Changing expert rankings and the retrospective comparator therefore matter alongside limited individual evidence.
Questions for authors
  1. How does prospective selection compare with a constant expert chosen exclusively from training users, under the same evaluation protocol?
  2. Do the central findings persist when the diagnostics are applied to an established personalized reward model rather than the fixed memory-correction predictor?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1VW4: New Evidence Addressing Soundness Concerns

We thank the reviewer for pointing out the questions about the baseline and the interpretation of the oracle result. We ran new checks for both. We also tested whether the selection result changes when we use features from an 8B reward model. We keep our responses in order of how consequential the concerns are.


Q1 [W2; Reviewer Q1]: Does choosing the baseline with hindsight explain the result?

The original comparison used the best common rule on the later evaluation window. We repeated it using a common rule chosen only from training users. The training and test users do not overlap. The two references choose the same rule in 9 of 10 seeds. With the training-selected reference, choosing a rule from each user's earlier window gives:

  • Main mixture: +0.12 percentage points, 95% CI [−0.67, 0.90].
  • Change-point mixture: −1.61 points, 95% CI [−2.32, −0.90].

Choosing the rule after seeing the later outcomes instead gives +5.81 and +4.09 points. We added the per-seed results to Appendix P.1, Table 28.

Key Takeaway. The choice of the common baseline does not explain the result. Earlier-window selection still shows no reliable improvement, and it performs worse in the change-point mixture.


Q2 [W2]: Why does even an oracle that knows the best earlier rule fail?

There are two different issues that need to be tested. First, can we identify the best rule from limited earlier choices, and second, will that rule still be best later? We now separate them using the known choice probabilities in our controlled histories. We divide each history into an earlier window A and a later window B. Three effects explain the gap between choosing with hindsight on B and choosing from A:

Source of the gap (points)StableMainChange-point
Optimism from seeing B's outcomes1.862.992.45
Cost of selecting from observed choices in A1.210.930.90
Cost of using A's truly best rule in B0.401.742.51
Total gap3.475.665.87

The parts add up exactly before rounding, using the same cross-fitted reference. Even when users' preferences never change, the gap is 3.47 points. The best rule can still differ between A and B because the candidate comparisons differ and the rules are often nearly tied.

When abrupt preference changes are common, the cost of using A's truly best rule in B rises from 0.40 to 2.51 points. The truly best rule differs across windows for 42% of users, compared with 15% in the stable-only setting. This also explains the oracle's −0.87-point result in Table 14: knowing the best rule for A does not guarantee that it is best for B. The revised Figure 4 and Table 29 show all three effects.

Key Takeaway. The prospective gap is not simply noisy rule selection. We now measure separately the effect of hindsight, limited earlier evidence, and a change in which rule performs best.


Q3 [W1; Reviewer Q2]: Why keep the predictor fixed? Does the result hold with reward-model features?

We are not claiming that existing personalized reward models fail. We instead test what the available history can justify about a particular user's memory. For that, we keep the shared predictor and response representation fixed across memory rules so that we can compare how they use the same history. Controlled histories also tell us whether a user's underlying preferences really changed and when. That information is used for evaluation and is hidden from the tested policies. Natural user logs do not provide the same ground truth.

We also repeated the selection test using features from Skywork-Reward-V2-Llama-3.1-8B on real histories. Against a common rule chosen from training users, earlier-window selection gains −0.75 points on Arena (53 test users) and +0.04 points on PRISM (488 test users). Neither result establishes an improvement.

Key Takeaway. The fixed predictor and controlled histories let us isolate the memory decisions in the paper. The 8B feature test provides another check. Testing complete PRM architectures would show how widely the findings apply. Note that it is not required to establish the results within our controlled setting.

We hope these results address the soundness concerns and clarify the scope of our findings. Modifications in the revised uploaded manuscript are marked in blue.

AuthorsOpenReview ↗

Gentle reminder for valuable feedback

Dear Reviewer 1VW4,

Thank you again for your review and time. As the discussion period is about to close, we wanted to gently follow up on our response and revised manuscript. We have now directly addressed the main concerns raised in your review:

  • Hindsight-selected baseline: We repeated the comparison using only disjoint training users. The conclusion remains unchanged (Table 28).
  • Oracle result in Table 14: We added a stable-only control and separated the three causes of the prospective-selection gap, including why the earlier-window oracle can fail (Figure 4, Table 29).
  • Reward-model validation: We repeated the selection test using 8B Skywork reward-model features on Arena, PRISM, and PENS, with the results and scope clearly reported (Table 30).

All additions are highlighted in blue in the revised manuscript. We believe we have addressed your concerns, particularly regarding soundness. We would greatly appreciate to get your valuable feedback. Please let us know if any specific issues remain. We would be happy to clarify it before the discussion closes. Thank you for your time and consideration.

Thanks for the detailed response, which partially addressed my concerns. I will increase my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.