acceptodds
Under review as a conference paper at ICLR 2027

Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling

Abstract

Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences. Personalized reward models (PRMs) address this by conditioning rewards on user-specific feedback, most commonly through in-context learning (ICL), where a user's historical comparisons are supplied as contextual preference pairs. However, we identify a key limitation of ICL-based PRMs: they fail to capture the preference relations conveyed by contextual pairs. To address this, we propose Preference-Aligned Test-Time Training (P-TTT), which explicitly encodes these relations into user-specific fast weights for personalized reward prediction. P-TTT introduces sequence-level update and apply operations to match the response-level granularity of preference feedback, together with a preference-aligned objective that directly uses pairwise preference relations to guide fast-weight adaptation. Notably, P-TTT is simple to implement and computationally efficient, updating fast weights within a single forward pass without inference-time backpropagation. Extensive experiments show that P-TTT more effectively captures historical preference relations and outperforms state-of-the-art methods by a large margin.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, Reward model, Personalization, Test time training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nWFiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether personalized reward models effectively use the preference relations expressed in users' historical comparisons. It proposes Preference-Aligned Test-Time Training (P-TTT), which updates MLP down-projection weights using chosen--rejected response representations and applies the resulting adaptation at the target response's final token. The update has a closed-form expression and requires no inference-time backpropagation. Experiments on three synthetic preference benchmarks and two backbones show consistent accuracy improvements. The problem is important, and the proposed inductive bias is reasonable. However, the experiments establish the performance gains more convincingly than the claimed explanation of ICL's limitations or the applicability of the theoretical analysis.

Critical strengths
  1. The method provides a concrete adaptation of TTT to personalized reward modeling. It uses response-level representations and an explicit chosen/rejected objective to compute user-specific fast-weight updates in closed form. This makes the adaptation directly relevant to the prediction task while avoiding inference-time backpropagation. The ablations in Table 2 support the usefulness of both design choices.
  2. The empirical improvements are consistent across the reported benchmarks. P-TTT achieves the highest mean accuracy on all three datasets with both backbones, with results averaged over three seeds and an average gain of 3.94 percentage points over the strongest baseline in each setting.
Critical weaknesses
  1. The theoretical justification does not establish the claimed behavior of the evaluated architecture. In Appendix B, Eq. (17) equates the final reward change to a linear readout of the adapted block's output change. This does not follow for an internal block followed by nonlinear Transformer layers and normalization; restricting adaptation to one block does not by itself resolve this issue, and the experiments adapt multiple blocks. Theorem 2 additionally assumes an identity value projection and exactly zero aggregate contribution from all non-matching keys (Appendix C). These results provide intuition for a restricted setting, but do not establish reward-margin antisymmetry or preference-improving updates in the actual model. This matters because the theory is presented as support for the central mechanism, rather than clearly separated as an idealized illustration.
  2. Generalization to unseen preference dimensions remains untested, limiting the evidence for the method's broader practical motivation. UF-P-2 and UF-P-4 use the same preference types for training and testing. PersonalLLM includes unseen users generated from the same reward-model family, and their results are pooled with those of seen users; new users therefore do not establish transfer to new preference dimensions. The current results demonstrate improvements within the evaluated preference distributions, but leave open whether the learned adaptation mechanism can use limited feedback to capture a type of preference absent from offline training.
Questions for authors
  1. Does Eq. (17) describe a linear surrogate or the actual reward network? Please state the assumptions needed for Theorems 1-2, including downstream nonlinear layers and the learned value projection. If the exact reward-margin antisymmetry is intended to apply to the implemented model, can you measure its deviation from Eq. (12) when all contextual labels are reversed? Otherwise, please restrict the theorem statements and interpretation accordingly.
  2. Could you evaluate generalization to unseen preference dimensions, for example by training on UF-P-2 (helpfulness and honesty) and testing on held-out preference types using only user history for adaptation? This would clarify whether P-TTT learns a transferable adaptation mechanism and retains its advantage over ICL and MRM beyond the training preference distribution.
  3. Since Eq. (23) conditions on each model's initially correct predictions, could you compare the flip rates of ICL and P-TTT on the same valid subset where both models are initially correct? Does the gap persist under this common-subset comparison?
  4. Could you evaluate using a survey pool disjoint from training, keeping the target examples and history length fixed?
6 comments
AuthorsOpenReview ↗

Kind Reminder and Summary of Our Response

Dear Reviewer nWFi,

Thank you again for your detailed and constructive comments. The discussion period is ending soon, so we wanted to kindly check whether our responses have addressed your concerns. In brief:

  • (W1 & Q1) We clarified the scope and assumptions of Theorems 1–2 in the revised version.
  • (W2 & Q2) We added evaluations on unseen users and an unseen preference dimension, where P-TTT achieves the best accuracy in both settings.
  • (Q3) We recomputed the flip rates on the common subset where both models are initially correct, and the gap between P-TTT and ICL persists on all datasets.
  • (Q4) We evaluated with a survey pool disjoint from training, and P-TTT still outperforms ICL on all three datasets.

We would be grateful if you could take a look, and we hope you might consider updating your score if your concerns have been resolved.

Best regards, The Authors

Thank you for the clarifications and additional experiments. While they address some of my concerns, two questions remain:

  • When the survey pool changes, P-TTT’s accuracy drops from 75.40% to 57.33% on UF-P-2, and its lead over ICL shrinks to 0.6 percentage points on UF-P-4. It still outperforms ICL, but these drops leave me concerned about how well it generalizes to new history pairs.

  • The unseen-preference experiment tests only truthfulness, while honesty, a closely related dimension, is included in training. This is a useful addition, but it leaves open whether P-TTT can adapt to preferences that differ more from those seen during training.

Based on these remaining concerns, I will maintain my current score for now.

AuthorsOpenReview ↗

Response to Follow-up Comments

Thank you for your follow-up. We are glad that our responses to W1 & Q1, Q3, and the unseen-user part of W2 & Q2 have addressed your concerns. We address the two remaining questions below.

Response to "When the survey pool changes, P-TTT’s accuracy drops from 75.40% to 57.33% on UF-P-2, and its lead over ICL shrinks to 0.6 percentage points on UF-P-4. It still outperforms ICL, but these drops leave me concerned about how well it generalizes to new history pairs."

Thank you for raising this concern. We attribute the drop to the small size of the survey pool in our setting (100 pairs), which makes it difficult for models to generalize to a different pool. We find that enlarging the survey pool substantially narrows the gap. We enlarge the training survey pool to 500 pairs and evaluate on a disjoint survey pool of 500 held-out pairs from the same source, all excluded from the training targets. The test target pairs and the history length (m = 4) are unchanged.

MethodUF-P-2UF-P-4
ICL50.6057.17
P-TTT64.4262.41

With a larger survey pool, P-TTT’s accuracy on unseen history pairs rises from 57.33% to 64.42% on UF-P-2 and from 59.82% to 62.41% on UF-P-4. Its lead over ICL widens to 13.82 and 5.24 percentage points on UF-P-2 and UF-P-4, respectively. This suggests that the drop observed with the 100-pair pool is largely due to the limited pool size, and that enlarging the pool substantially mitigates it.

We also note that a shared survey pool is the standard protocol in prior work (CoPL [1], SPL [2], PLUS [3], VPL [4], MRM [5]). It matches practical deployment: a personalized system typically collects feedback from every user, including new ones, on the same fixed set of survey questions. The generalization required in practice is therefore to new users, which is held out in our evaluation (Section 5.3).

References:

[1] Choi et al. CoPL: Collaborative Preference Learning for Personalizing LLMs. EMNLP 2025.

[2] Kim & Kim. Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.

[3] Nam et al. Learning to Summarize User Information for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.

[4] Poddar et al. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning. NeurIPS 2024.

[5] Cai et al. One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment. SIGIR 2026.

Response to “The unseen-preference experiment tests only truthfulness, while honesty, a closely related dimension, is included in training. This is a useful addition, but it leaves open whether P-TTT can adapt to preferences that differ more from those seen during training.”

We thank the reviewer for this suggestion. To test whether P-TTT can adapt to preferences that differ more from those seen in training, we add a harder setting on UF-P-4. We remove honesty, train only on helpfulness and instruction following, and test on truthfulness.

Train preferencesTest preference (unseen)ICLP-TTTΔ
Help + Hon + IFTruth49.0456.49+7.45
Help + IFTruth43.0353.12+10.09

Without honesty in training, ICL drops below chance (43.03%), while P-TTT still achieves 53.12%. Its margin over ICL also grows from +7.45 to +10.09. This suggests that P-TTT remains effective even when the test preference differs substantially from those seen during training.

Reviewer_TrNBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper studies personalized reward models and claims to identify a key limitation of in-context learning approaches: although they use a user’s historical preference pairs, their predictions barely change when the chosen and rejected labels are swapped, suggesting they do not capture preference direction. To address this, the authors propose P-TTT, which encodes each user’s historical preferences into a small set of fast weights through a simple, backpropagation-free test-time update, removing the need to keep the history in the context. On UF-P-2, UF-P-4 and PersonalLLM with two small backbones, P-TTT outperforms eight baselines while running faster than ICL.

Critical strengths
  1. The paper points out a strong and interesting finding that user's historical preference pairs actually help little for predictions.
  2. The paper proposes a lightweight and effective test time training method, and achieved performance improvement.
  3. The paper is easy to follow and logically clear.
Critical weaknesses
  1. Small backbones undermine the core claim. The conclusion that ICL fails to capture preference relations rests on only 0.5B and 1B models, where the ICL baseline appears to overfit (about 97% train vs. 53% test accuracy). Experiments with larger models are needed as ICL tends to improve a lot with model size. I'm not asking for over 100B models, but so far 1B is too small for such strong claim. You can only claim such strong finding either with extreme large model results (I don't actually recommend as normally it's not feasible) OR a scaling trend analysis (e.g. if from 0.5B to 32B, the improvement of ICL is marginal)
  2. The evaluation is only based on synthetic users and requires preference labels of the same user, which is far from real-world personalization. It's unclear whether P-TTT applies to more realistic setting, where user profile is more diverse (w/o preference pairs) and non-synthetic (e.g. Personalized RewardBench).
  3. Several baselines score near 50% on datasets where their original papers reported much more gains, e.g. VPL. (need more details on baseline reimplementation, current results seem contradict with baseline papers)
  4. The higher flip rate largely follows by construction: the P-TTT update is linear in the preference labels, so swapping them mechanically reverses the induced reward change, as Theorem 1 itself shows. Therefore, a high flip rate shows that the model is sensitive to the labels, not that it captures meaningful preference relations, and comparing it with ICL (which has no such built-in property) is not an informative test. (no need to stick to flip rate as you already have other direct results)
Questions for authors

please refer to the weaknesses, the first three are very major issues, the fourth is minor.

6 comments
AuthorsOpenReview ↗

Kind Reminder and Summary of Our Response

Dear Reviewer TrNB,

Thank you again for your careful reading and valuable suggestions. The discussion period is ending soon, so we wanted to kindly check whether our responses have addressed your concerns. In brief:

  • (W1) We added experiments on Qwen3-8B and GPT-5.6-terra, where ICL remains limited even on much larger models while P-TTT keeps its large advantage.
  • (W2) We clarified that our setting follows the standard protocol in personalized reward modeling, and we outlined how P-TTT could extend to Personalized RewardBench.
  • (W3) We added the implementation details and key hyperparameters of all baselines to the revised version.
  • (W4) Beyond flip rate and accuracy, we further added personalized generation experiments, where P-TTT outperforms ICL.

We would be grateful if you could take a look, and we hope you might consider updating your score if your concerns have been resolved.

Best regards, The Authors

Thank you for your response. W1 and W4 have been well addressed, and my remaining concerns are regarding W2 and W3.

Regarding W2: More recent personalized reward benchmarks focus on incorporating persona information as input (e.g., Personalized RewardBench [1] and PersonalRewardBench [2]), extending the scope of personalized reward modeling beyond pairwise preference inputs. This setting is also more representative of real-world scenarios, where user profiles are stored in text instead preference pairs. I understand that this may be beyond the scope of your current design, since your model relies exclusively on pairwise preferences as personalization signals. However, given your argument regarding the inherent limitations of ICL in understanding user context, I believe this broader personalization setting deserves consideration, at least as a discussion of the limitations of your approach.

Regarding W3: Both SPL and VPL report accuracies above 62% on UF-P-2 and 57% on UF-P-4 using Llama-3.2-3B, whereas your results for these baselines are around 50% using Llama-3.2-1B. I understand that your experiments primarily focus on 0.5B and 1B models, but I wonder whether the difference in model scale alone can explain such a substantial performance gap. While I may be mistaken, the current evidence is not sufficiently convincing to establish the correctness of the baseline implementations. Nevertheless, I appreciate the implementation details provided in the appendix and would welcome further clarification on this discrepancy.

[1] Personalized RewardBench: Evaluating reward models with human aligned personalization. COLM 2026 [2] SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs. ACL 2025 Main

AuthorsOpenReview ↗

Response to Follow-up Comments (W2 & W3)

We thank the reviewer for the follow-up and are glad that W1 and W4 have been addressed. We respond to the remaining concerns below.

Response to "Recent benchmarks such as Personalized RewardBench and PersonalRewardBench use textual persona information as input, which better reflects real-world scenarios. Given the paper's argument about the limitations of ICL in understanding user context, this broader setting deserves consideration, at least as a discussion of the limitations of P-TTT."

We thank the reviewer for this valuable suggestion. In the revised paper, we have added a discussion of the limitations of P-TTT to Section 6: its current design uses pairwise preference feedback as the personalization signal, and extending it to settings with persona information is a promising direction for future work. In addition, the Appendix K discusses how P-TTT can be extended to this setting, for example, by replacing the chosen-vs-rejected contrast with a contrast between the user’s own profile items and a population background.

Response to "SPL and VPL report accuracies above 62% on UF-P-2 and 57% on UF-P-4 with Llama-3.2-3B, while the reproduced results are around 50% with Llama-3.2-1B. It is unclear whether model scale alone can explain such a large gap, and the current evidence is not sufficient to establish the correctness of the baseline implementations."

Thank you for raising this concern. For data construction, we follow more recent work rather than the early VPL setup (also used by SPL). This difference may account for the gap in absolute numbers. Below, we (1) detail how the two protocols differ, and show that under the same protocol our reproduced results closely match those reported in recent work; (2) show that later work also reports only marginal gains of VPL over the non-personalized BTL; and (3) report results with Llama-3.2-3B, showing that model scale is not the cause of the gap.

1. We follow the data construction protocol of more recent work, which differs from VPL/SPL.

The two protocols differ mainly in how controversial preference pairs are selected.

In UF-P-2, one user prefers helpfulness and the other prefers honesty. The UF-P-2 data built with the official VPL code keeps many non-personalized pairs, i.e., pairs on which both users give the same label (about 48%). These pairs can be judged without the user’s past preference pairs. As a result, in SPL (Llama-3.2-3B), the non-personalized BTL already reaches 62.25%, while VPL and SPL improve on it only marginally (62.37% and 63.28%).

Following the recent work CoPL [1], we remove these pairs and keep only those on which the two users disagree, so that every sample requires user-specific preference information to be judged correctly. Under this protocol, our results closely match those reported by [1]: VPL obtains 50.16 / 50.24 (vs. 50.26 in CoPL), and BTL obtains 49.88 / 49.92 (vs. 49.86).

Notably, on UF-P-4, where both protocols define controversy over the same four attributes (so no fully agreed pairs remain), our VPL results (57.73 / 58.57) are on par with those reported in SPL (57.03 / 57.14 with Llama-3.2-3B / 3.1-8B).

This shows that our implementation is consistent with theirs, and that the UF-P-2 gap stems from the data. In the revised paper, we have expanded Appendix I with the full dataset construction details.

2. Later work also finds only marginal gains of VPL over BTL.

Recent papers that use VPL as a baseline consistently report small differences between VPL and the non-personalized BTL:

PaperBackboneUF-P-2: BTL → VPLUF-P-4: BTL → VPL
SPL [2]Llama-3.2-3B62.25 → 62.37 (+0.12)57.07 → 57.03 (−0.04)
SPL [2]Llama-3.1-8B62.59 → 62.66 (+0.07)57.40 → 57.14 (−0.26)
PLUS [3]Qwen2.5-0.5B58.12 → 58.37 (+0.25)53.53 → 54.20 (+0.67)
CoPL [1]gemma-2b-it49.86 → 50.26 (+0.40)56.15 → 56.04 (−0.11)

SPL attributes VPL’s limited personalization to posterior collapse: the model fails to make effective use of the user-specific latent variable.

3. Additional experiments on Llama-3.2-3B.

Following the reviewer’s suggestion, we further evaluate the baselines with Llama-3.2-3B:

MethodUF-P-2UF-P-4
VPL50.2658.55
SPL50.1359.64

Even with the same 3B backbone, VPL and SPL still achieve limited performance, which indicates that model scale does not explain the gap.

References:

[1] Choi et al. CoPL: Collaborative Preference Learning for Personalizing LLMs. EMNLP 2025.

[2] Kim & Kim. Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.

[3] Nam et al. Learning to Summarize User Information for Personalized Reinforcement Learning from Human Feedback. ICLR 2026.

Reviewer_KuZVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that reward models using in-context learning often overlook which responses users prefer in their historical feedback. They proposes P-TTT, which uses these preference pairs to update user-specific fast weights in a single forward pass without inference-time backpropagation. Experiments show improved personalized preference prediction over existing baselines.

Critical strengths
  1. The paper effectively adapts test-time training(TTT) to personalized reward models by accounting for the structure of preference feedback. In particular, deriving a learning signal from preference flips is an appealing aspect of the approach.

  2. The equations clearly explain how the method works, and the theoretical analysis is easy to follow. I think this is the paper’s main strengths.

Critical weaknesses
  1. Evaluation is limited to very small language models. (0.5B-1B)

    In my experience, small models struggle to use user preferences effectively. Past user preference data which unrelated to the current question can act as noise that makes understanding task or user preference harder.

    Larger models may be better at using examples to understand the user’s preferences, so this problem may be less of an issue. However, P-TTT is tested only on very small models like 0.5B, where updating the weights at test-time may give it a large advantage over the baselines. It is unclear whether P-TTT’s large advantage would remain with larger models.

  2. Higher reward-model accuracy does not guarantee better personalized generation.

    Many PRMs papers argue that better reward-model accuracy leads to better personalized alignment and generation. But better accuracy does not necessarily mean the model will generate answers that users really prefer.

    PRMs can be used for post-training or reward-guided decoding(RGD). However, both has challenges. Adding personalization makes post-training more difficult. Also, a reward model trained to score complete answers may not work well when used for RGD.

    I would like to see more explanation or evidence of how P-TTT helps generate answers that better match users’ preferences compared with existing PRMs or SFT methods such as Nextquill[1].

    [1] Zhao, Xiaoyan, et al. "Nextquill: Causal preference modeling for enhancing llm personalization." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. In section 3, how is the ICL baseline built?

    Section 3 discusses the limitations of ICL-based PRMs, but it is unclear how the ICL baseline was built and trained. Is it a standard reward model trained with past preference examples in its input, or is it based on a method such as PLUS?

  2. How robust is P-TTT to label-flip noise?

    P-TTT directly uses chosen and rejected labels to update its weights. However, users may sometimes provide incorrect labels. Given how P-TTT works, I’m concerned that even a few wrong labels could have too much influence on its predictions. How well does P-TTT handle this label-flip noise compared with existing PRMs?

4 comments
AuthorsOpenReview ↗

Kind Reminder and Summary of Our Response

Dear Reviewer KuZV,

Thank you again for your thoughtful review. The discussion period is ending soon, so we wanted to kindly check whether our responses have addressed your concerns. In brief:

  • (W1) We added experiments on Qwen3-8B and GPT-5.6-terra, showing that ICL remains limited even at larger scale, while P-TTT’s advantage holds.
  • (W2) We added personalized generation experiments with Best-of-N sampling, and P-TTT outperforms the ICL-based PRM on all 12 metrics.
  • (Q1) We clarified that the ICL baseline is a reward model that takes the user's historical preference pairs as context and is trained with Eq. (3).
  • (Q2) We added label-flip noise experiments, where P-TTT outperforms ICL and MRM at every noise level.

We would be grateful if you could take a look, and we hope you might consider updating your score if your concerns have been resolved.

Best regards, The Authors

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer KuZV,

We are sorry to follow up again and fully understand how busy this period is. With the discussion period closing soon, we would be very grateful if you could take a brief look at our responses and the revised paper, where we have added the experiments on larger models (Qwen3-8B, GPT-5.6-terra), personalized generation, and label-flip noise.

If any concern remains, we would be glad to clarify it before the deadline. If our responses have addressed your concerns, we would appreciate it if you could consider updating your score.

Best regards, The Authors

Thank you for the detailed responses and additional experiments. These address most of my concerns.

I find the generation results encouraging. NextQuill directly fine-tunes the generator with a cross-entropy-based objective, which may give it an advantage on reference-based metrics such as ROUGE and BERTScore. Given this difference, P-TTT performs well using its reward model to select responses from a frozen generator.

Also, it would be useful to evaluate the generated responses using an LLM judge provided with the user’s preference history. This could help assess how well the responses reflect individual preferences beyond similarity to the reference responses. However, I do not think this is necessary for this rebuttal.

Since the authors have addressed all my questions and concerns, I am raising my score from 2 to 3.

Thank you for your thoughtful feedback and for raising your score from 2 to 3. We are glad that our responses and additional experiments addressed your concerns, and we appreciate your encouraging assessment of P-TTT’s generation results. As you suggested, we will add LLM judge scores as a metric in the personalized generation experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.