- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether personalized reward models effectively use the preference relations expressed in users' historical comparisons. It proposes Preference-Aligned Test-Time Training (P-TTT), which updates MLP down-projection weights using chosen--rejected response representations and applies the resulting adaptation at the target response's final token. The update has a closed-form expression and requires no inference-time backpropagation. Experiments on three synthetic preference benchmarks and two backbones show consistent accuracy improvements. The problem is important, and the proposed inductive bias is reasonable. However, the experiments establish the performance gains more convincingly than the claimed explanation of ICL's limitations or the applicability of the theoretical analysis.
Critical strengths
- The method provides a concrete adaptation of TTT to personalized reward modeling. It uses response-level representations and an explicit chosen/rejected objective to compute user-specific fast-weight updates in closed form. This makes the adaptation directly relevant to the prediction task while avoiding inference-time backpropagation. The ablations in Table 2 support the usefulness of both design choices.
- The empirical improvements are consistent across the reported benchmarks. P-TTT achieves the highest mean accuracy on all three datasets with both backbones, with results averaged over three seeds and an average gain of 3.94 percentage points over the strongest baseline in each setting.
Critical weaknesses
- The theoretical justification does not establish the claimed behavior of the evaluated architecture. In Appendix B, Eq. (17) equates the final reward change to a linear readout of the adapted block's output change. This does not follow for an internal block followed by nonlinear Transformer layers and normalization; restricting adaptation to one block does not by itself resolve this issue, and the experiments adapt multiple blocks. Theorem 2 additionally assumes an identity value projection and exactly zero aggregate contribution from all non-matching keys (Appendix C). These results provide intuition for a restricted setting, but do not establish reward-margin antisymmetry or preference-improving updates in the actual model. This matters because the theory is presented as support for the central mechanism, rather than clearly separated as an idealized illustration.
- Generalization to unseen preference dimensions remains untested, limiting the evidence for the method's broader practical motivation. UF-P-2 and UF-P-4 use the same preference types for training and testing. PersonalLLM includes unseen users generated from the same reward-model family, and their results are pooled with those of seen users; new users therefore do not establish transfer to new preference dimensions. The current results demonstrate improvements within the evaluated preference distributions, but leave open whether the learned adaptation mechanism can use limited feedback to capture a type of preference absent from offline training.
Questions for authors
- Does Eq. (17) describe a linear surrogate or the actual reward network? Please state the assumptions needed for Theorems 1-2, including downstream nonlinear layers and the learned value projection. If the exact reward-margin antisymmetry is intended to apply to the implemented model, can you measure its deviation from Eq. (12) when all contextual labels are reversed? Otherwise, please restrict the theorem statements and interpretation accordingly.
- Could you evaluate generalization to unseen preference dimensions, for example by training on UF-P-2 (helpfulness and honesty) and testing on held-out preference types using only user history for adaptation? This would clarify whether P-TTT learns a transferable adaptation mechanism and retains its advantage over ICL and MRM beyond the training preference distribution.
- Since Eq. (23) conditions on each model's initially correct predictions, could you compare the flip rates of ICL and P-TTT on the same valid subset where both models are initially correct? Does the gap persist under this common-subset comparison?
- Could you evaluate using a survey pool disjoint from training, keeping the target examples and history length fixed?
6 comments
Thank you for the clarifications and additional experiments. While they address some of my concerns, two questions remain:
-
When the survey pool changes, P-TTT’s accuracy drops from 75.40% to 57.33% on UF-P-2, and its lead over ICL shrinks to 0.6 percentage points on UF-P-4. It still outperforms ICL, but these drops leave me concerned about how well it generalizes to new history pairs.
-
The unseen-preference experiment tests only truthfulness, while honesty, a closely related dimension, is included in training. This is a useful addition, but it leaves open whether P-TTT can adapt to preferences that differ more from those seen during training.
Based on these remaining concerns, I will maintain my current score for now.
Response to Follow-up Comments
Thank you for your follow-up. We are glad that our responses to W1 & Q1, Q3, and the unseen-user part of W2 & Q2 have addressed your concerns. We address the two remaining questions below.
Response to "When the survey pool changes, P-TTT’s accuracy drops from 75.40% to 57.33% on UF-P-2, and its lead over ICL shrinks to 0.6 percentage points on UF-P-4. It still outperforms ICL, but these drops leave me concerned about how well it generalizes to new history pairs."
Thank you for raising this concern. We attribute the drop to the small size of the survey pool in our setting (100 pairs), which makes it difficult for models to generalize to a different pool. We find that enlarging the survey pool substantially narrows the gap. We enlarge the training survey pool to 500 pairs and evaluate on a disjoint survey pool of 500 held-out pairs from the same source, all excluded from the training targets. The test target pairs and the history length (m = 4) are unchanged.
| Method | UF-P-2 | UF-P-4 |
|---|---|---|
| ICL | 50.60 | 57.17 |
| P-TTT | 64.42 | 62.41 |
With a larger survey pool, P-TTT’s accuracy on unseen history pairs rises from 57.33% to 64.42% on UF-P-2 and from 59.82% to 62.41% on UF-P-4. Its lead over ICL widens to 13.82 and 5.24 percentage points on UF-P-2 and UF-P-4, respectively. This suggests that the drop observed with the 100-pair pool is largely due to the limited pool size, and that enlarging the pool substantially mitigates it.
We also note that a shared survey pool is the standard protocol in prior work (CoPL [1], SPL [2], PLUS [3], VPL [4], MRM [5]). It matches practical deployment: a personalized system typically collects feedback from every user, including new ones, on the same fixed set of survey questions. The generalization required in practice is therefore to new users, which is held out in our evaluation (Section 5.3).
References:
[1] Choi et al. CoPL: Collaborative Preference Learning for Personalizing LLMs. EMNLP 2025.
[2] Kim & Kim. Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.
[3] Nam et al. Learning to Summarize User Information for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.
[4] Poddar et al. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning. NeurIPS 2024.
[5] Cai et al. One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment. SIGIR 2026.
Response to “The unseen-preference experiment tests only truthfulness, while honesty, a closely related dimension, is included in training. This is a useful addition, but it leaves open whether P-TTT can adapt to preferences that differ more from those seen during training.”
We thank the reviewer for this suggestion. To test whether P-TTT can adapt to preferences that differ more from those seen in training, we add a harder setting on UF-P-4. We remove honesty, train only on helpfulness and instruction following, and test on truthfulness.
| Train preferences | Test preference (unseen) | ICL | P-TTT | Δ |
|---|---|---|---|---|
| Help + Hon + IF | Truth | 49.04 | 56.49 | +7.45 |
| Help + IF | Truth | 43.03 | 53.12 | +10.09 |
Without honesty in training, ICL drops below chance (43.03%), while P-TTT still achieves 53.12%. Its margin over ICL also grows from +7.45 to +10.09. This suggests that P-TTT remains effective even when the test preference differs substantially from those seen during training.
Kind Reminder and Summary of Our Response
Dear Reviewer nWFi,
Thank you again for your detailed and constructive comments. The discussion period is ending soon, so we wanted to kindly check whether our responses have addressed your concerns. In brief:
We would be grateful if you could take a look, and we hope you might consider updating your score if your concerns have been resolved.
Best regards, The Authors