- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether detecting preference change is sufficient to justify memory adaptation in personalized reward models. Using a shared Bradley–Terry predictor, it compares retention, discounting, multiscale memory, and event-triggered reset. Experiments on synthetic and natural histories distinguish population-level temporal signal from individual actionability, change detection from intervention choice, and adaptive memory allocation from evidence of individual preference change.
Critical strengths
-
The paper addresses a meaningful evaluation problem and clearly distinguishes detecting preference change from deciding whether and how memory should adapt.
-
The prospective evaluation, selection-optimism decomposition, and shared-alarm comparisons provide useful controls against misleading conclusions about adaptation.
-
The experiments cover several temporal regimes and natural datasets. The authors transparently report negative findings and distinguish prespecified results from exploratory analyses.
Critical weaknesses
- The information paradox mixes limited individual evidence with changes in the best memory rule across windows. Eeven knowing the true best earlier-window expert can hurt later performance, while longer histories yield positive selection gains. These results warrant a clearer separation between estimation difficulty and temporal transfer failure.
- The strongest findings rely on synthetic histories and a linear predictor over frozen representations. Natural PRISM and Arena results are mostly inconclusive, PENS uses a constructed click surrogate, and Arena splices represent switches between users. The checkpoint analyses do not repeat the full selection or attribution tests, limiting validation of the central claims in realistic PRMs.
- The conceptual contribution beyond established selection bias, forgetting, and change-detection principles needs sharper justification.
Questions for authors
- Can the prospective-selection analysis separately quantify finite-sample estimation error and changes in the true best expert between windows? Would the conclusion hold when both windows lie within the same stable post-change regime?
- Does the population-dependent shortening persist when memory is measured using operative lag contributions or prediction changes after removing older observations?
2 comments
Thanks for your efforts. I maintain my positive rating.
Response to Reviewer i6Hp: Resolving Technical Concerns and Clarifying Scientific Novelty
We thank the reviewer for the careful comments. We conduct two tests that separate why earlier-window rule selection can fail and show that shorter memory for stable users changes actual predictions. We respond in order of the implications of the concerns.
Q1 [W1; Reviewer Q1]: Does selection fail because of noise or because the best rule changes?
Both can matter. Using the known choice probabilities in our controlled data, we divide the hindsight–prospective gap into 3 parts (Fig. 4 / Table 29.):
Even when preferences never change, the gap is 3.47 points. Different candidate comparisons and nearly tied rules can change which rule is best. The truly best rule differs between windows for 15% of stable users and 42% in the change-point mixture. The cost of using the earlier best rule rises from 0.40 to 2.51 points. This helps explain why the earlier-window oracle loses 0.87 points against the common rule in Table 14.
The stable-only experiment confirms that the problem exists even without preference change. The three-part analysis then shows how much comes from limited evidence and how much from the best rule changing across windows. Our longer-history experiments also show that more observations can improve selection.
Key Takeaway. We now separate and measure the reasons behind the Information Paradox. Limited evidence can make rule selection unreliable even for stable users, while changes in the best rule create an additional challenge.
Q2 [Reviewer Q2]: Does shorter memory actually affect predictions?
Yes. For the same stable users, the operative mean lag is 29.5 steps with a stable-trained controller and 18.4 with a change-point-trained controller. Nominal memory shows the same pattern (29.2 vs. 15.7).
We also removed older choices. Keeping trained controller weights fixed, we recalculated its memory and inputs from the shorter history. Removing choices older than 16 steps changes the predicted probability of the observed choice by 12.3 points for the stable-trained controller versus 7.8 for the change-trained controller. The paired difference is 4.57 points [3.92, 5.21] across five seeds. The same ordering holds at 8- and 32-step cutoffs and without the history-length input (Fig. 9; Appendix P.2).
Key Takeaway. The shortening changes predictions. The change-trained controller relies less on older choices, even for stable users.
Q3 [W2]: Why use controlled histories and a fixed predictor rather than an existing personalized reward model?
We do not claim existing PRMs fail. We keep the predictor and representations fixed across memory rules so differences reflect how history is used. Controlled histories tell us whether and when preferences truly changed, while hiding that information from the tested policies. Note that natural user logs do not provide such ground truth.
We also repeated the selection test using Skywork-Reward-V2-Llama-3.1-8B features on real Arena and PRISM histories. Against common rules chosen from separate training users, earlier-window selection gains -0.75 points on Arena and +0.04 points on PRISM. Neither establishes an improvement (Appendix P.3 and Table 30). This experiment uses features extracted from an existing reward model inside our fixed predictor. It does not test a separately trained personalized reward model with its own learned user representation.
Key Takeaway. Keeping the predictor fixed lets us isolate the effect of memory. Testing a separately trained personalized reward model would tell us whether the findings also hold in another architecture. It does not change what our controlled experiments establish.
Q4 [W3]: What scientific insight is new?
A temporal PRM still faces a chain of decisions: Does this user's history justify adapting? If so, should we retain, discount, or reset memory? Does the resulting memory reflect this user's evidence or is it a population pattern? What we see is that evidence for one decision does not answer the next. This insight does not follow from established results in selection bias, forgetting, and change detection. Our intervention test shows why. Recent evidence beats old evidence, yet keeping old choices helps by 2.33 points. Under the same alarms, reset's advantage over fast discount shifts from -0.28 at zero delay to +1.04 points at eight steps.
Key Takeaway. A change signal does not identify whom to adapt. Knowing whom does not determine how. And a changed memory does not imply individual change. Additional experiments and explanations in the revised manuscript are marked in blue.