acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Personalization via DPO with Latent Reward Features

Abstract

Standard preference alignment of language models (LMs) relies on a single reward model (RM), overlooking the diversity of human preferences. While recent work on personalized RMs address this by learning reward features—representing shared latent preference dimensions—together with user-specific weights, updating the alignment objective to these personalized weights may still require repeatedly performing costly preference optimization for each user. In this paper, we propose TPD (Test-time Personalized Decoding), a framework for on-the-fly personalization of the LM that avoids model retraining for each user. TPD first learns a set of shared feature-level policies through a DPO-like objective, bypassing the explicit learning of reward features. At test-time, TPD then balances these policy logits using weights adapted to the user's preferences, following a multi-objective decoding approach. Importantly, we show that TPD naturally extends to a broad class of alignment objectives based on -divergences. Experiments on synthetic and real-world datasets demonstrate effective steering of generation toward maximizing individual user satisfaction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Personalization, Test-time Alignment, Reward Models, Direct Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7JwwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TPD, which jointly learns a shared policy basis and user-specific weights from preference data. For new users, it fits only the weights and combines the policies during decoding, without further model fine-tuning. The method extends to several f-divergence objectives and is evaluated on synthetic and human preference datasets, with stronger gains in the synthetic setting.

Critical strengths
  1. Adapting only a small weight vector while keeping the policy basis frozen is a practical way to personalize generation without maintaining a separate model for each user.

  2. The controlled synthetic experiments show promising gains on unseen users with just eight preference examples. This supports the approach when user preferences follow the assumed shared latent structure.

Critical weaknesses
  1. LoRe already derives joint policy-basis learning and adaptation of new users’ weights, while MOD provides the decoding framework. The contribution beyond combining these methods seems limited, and the experiments do not show a consistent benefit from the added f-divergence choices.

  2. The real user results do not establish a clear personalization benefit. TPD averages 53.0% versus 52.3% for LoRe+BoN, with mixed results on unseen users and only one training seed. A comparison against uniform or shuffled user weights would help isolate the value of personalization.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

W1. Methodological Novelty and Benefit from f-divergence choices.

We thank you for your helpful feedback and careful reading. We would like to first clarify the novelty of the proposed approach and its importance for LM personalization. The central novelty of our approach is to establish a connection between reward-based multi-objective approaches (in particular, MOD) and reward feature learning approaches (in particular, LoRe). Importantly, to the best of our knowledge, as described in the related work in our paper (see section Multi-objective (MO) Test-time Alignment and Table 1), we are the first to enable multi-objective alignment with unknown objective dimensions and user weights, by learning them from unstructured preference examples (holistic pairwise comparisons). In doing so, our framework enables efficient test-time alignment to realistic human preferences. Furthermore, we realize this connection beyond a naive plug-in of LoRe and MOD, that would induce performing costly preference optimization with each reward features after learning it, by leveraging and extending to f-divergences LoRe’s formulation of DPO with latent reward features.

We also agree that further illustrating the interest of this extension to f-divergence could be valuable. Therefore, we made the following addition in the revised paper:

  • We include forward KL to enlarge the set of possible $f$-divergences and to fully study the standard spectrum from Reverse KL to Forward KL.
  • We provide in Figure 5 (included in the updated paper and presented in tabular form below with Table R1) a visual comparison of TPD and DPO across the four divergence choices on all datasets.Crucially, while performance decreases from reverse KL to forward KL on PersonalLLM and RP for both DPO and TPD, the opposite trend is observed on PRISM. This suggests that different divergences offer complementary benefits depending on the dataset, as also observed in prior work (Wang et al., 2024a; Go et al., 2023). We hypothesize that the advantage of forward KL (and $\alpha$-divergence) over reverse KL (and JSD) on PRISM stems from its higher level of noise (computed as the proportion of preference labels disagreeing with the user oracle, i.e., LLM judge with user profile). Specifically, reverse KL and JSD induce substantially larger preference margins (i.e., $\log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}$) in DPO (see Figure 10 in Appendix E.2), which potentially amplifies errors on mislabeled pairs.

Table R1: Win rate (%) against the base model per $f$-divergence. Tabular form of Figure 5 in the paper. Bold is the best divergence and $\underline{\text{underlined}}$ is the second best.

DatasetNoise (%)Methodreverse KLJSD$\alpha$-div. ($\alpha=0.5$)forward KL
PersonalLLM ($\gamma=0.01$)0.0TPD72.9±1.4$\underline{70.1\pm1.4}$62.8±1.559.3±1.6
DPO$\underline{\mathit{63.6\pm1.5}}$65.8±1.558.0±1.657.7±1.6
Community Alignment8.8±0.3TPD$\underline{53.8\pm3.0}$53.1±3.151.9±3.155.0±3.0
DPO$\underline{\mathit{54.6\pm3.0}}$58.6±3.151.4±3.352.8±3.3
Roleplay11.6±0.3TPD57.9±3.454.8±3.4$\underline{55.0\pm3.5}$48.4±3.3
DPO55.3±3.5$\underline{\mathit{51.6\pm3.6}}$49.9±4.251.1±4.0
PRISM21.4±0.3TPD47.2±1.845.8±1.850.6±1.8$\underline{50.5\pm1.8}$
DPO45.1±1.944.7±1.8$\underline{\mathit{47.5\pm2.0}}$49.7±2.0
AuthorsOpenReview ↗

W2. Clear personalization benefit on real users

We agree that further isolating the contribution of personalization on real users would be valuable. To this end, we added a comparison of TPD with adapted weights against TPD with uniform weights (TPD-uniform) across all datasets and divergence choices, in terms of both win rate ratio and preference accuracy. The results are presented in Table R5. On PersonalLLM (synthetic), TPD clearly improves over TPD-uniform across both metrics. On Roleplay, PRISM, and Community Alignment, TPD shows consistent (but minor) improvements over TPD-uniform for Reverse KL and JSD on both metrics, while sometimes underperforming with Forward KL and α-div.

Table R5a: Win rate ratio (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).

DatasetMethodreverse KLJSDα-div. (α=0.5)forward KL
Community AlignmentTPD-uniform53.5±3.052.7±3.056.2±3.057.6±3.0
TPD53.8±3.0 (+0.3)53.1±3.1 (+0.4)51.9±3.1 (-4.3)55.0±3.0 (-2.6)
RoleplayTPD-uniform57.5±3.353.3±3.450.5±3.553.4±3.5
TPD57.9±3.4 (+0.4)54.8±3.4 (+1.5)55.0±3.5 (+4.5)48.4±3.3 (-5.0)
PRISMTPD-uniform47.1±1.844.5±1.751.5±1.950.1±1.8
TPD47.2±1.8 (+0.1)45.8±1.8 (+1.3)50.6±1.8 (-0.9)50.5±1.8 (+0.4)
PersonalLLM (γ=0.01)TPD-uniform65.4±1.565.7±1.558.5±1.655.9±1.6
TPD72.9±1.4 (+7.5)70.1±1.4 (+4.4)62.8±1.5 (+4.3)59.3±1.6 (+3.4)

Table R5b: Preference accuracy (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).

DatasetMethodreverse KLJSDα-div. (α=0.5)forward KL
Community AlignmentTPD-uniform64.8±2.065.0±2.060.7±2.159.3±2.1
TPD66.0±2.0 (+1.2)65.8±2.0 (+0.8)64.4±2.1 (+3.7)62.8±2.1 (+3.5)
RoleplayTPD-uniform73.0±1.772.6±1.771.4±1.766.1±1.8
TPD75.5±1.6 (+2.5)75.1±1.6 (+2.5)72.3±1.7 (+0.9)66.1±1.8 (±0.0)
PRISMTPD-uniform61.5±1.461.1±1.460.7±1.459.5±1.4
TPD62.0±1.4 (+0.5)61.3±1.4 (+0.2)60.5±1.4 (-0.2)59.5±1.4 (±0.0)
PersonalLLM (γ=0.01)TPD-uniform74.5±1.473.1±1.471.5±1.472.4±1.4
TPD86.2±1.1 (+11.7)84.6±1.1 (+11.5)80.8±1.2 (+9.3)77.7±1.3 (+5.3)

This is consistent with the results reported in the paper, showing more modest personalization gains of TPD for simulated personas (Roleplay) and real users (PRISM, Community Alignment), compared to the clear gains observed on PersonalLLM, where users are simulated with deterministic utility functions. Given that TPD remains the best personalization method across all 7 implemented baselines on average in the real user experiments (Table 3 in the paper), we attribute those limited gains to the inherent difficulty of capturing real-user preferences that are scarce, dynamically evolving, and potentially context-dependent. This points to sequential, on-policy adaptation of user weights as an interesting future direction. We have added this discussion to the Conclusion of the revised paper. Table R5a-b are also included in Appendix E.2.3.

Thank you again for your feedback and constructive suggestions. We hope that our clarifications and additional experiments alleviate your concerns, and we remain available to answer any remaining questions or follow-ups during the rest of the discussion period

Thank you for running the additional experiments and for clarifying how TPD avoids a separate second stage of policy optimization compared with a naive combination of LoRe and MOD.

However, the new results in Table R5 reinforce my main concern about personalization on real users. On both real user datasets, adapting user weights gives almost no win rate gain over TPD with uniform weights under reverse KL and JSD, and even hurts performance under alpha divergence and forward KL. In fact, on both Community Alignment and PRISM, the highest win rate in Table R5a is mostly achieved by uniform weights, and standard DPO reaches 58.6% on Community Alignment in Table R1.

This also affects the interpretation of f divergences in W1. Although forward KL and alpha divergence improve overall win rates on noisier datasets in Table R1, Table R5a shows that personalized weights mostly match or underperform uniform weights under those divergences, indicating that the gain comes from regularizing the shared basis.

Overall, while the method works well on synthetic preferences, the empirical evidence for personalization on real human preferences remains weak, so I will maintain my current score.

AuthorsOpenReview ↗

TPD adapts its degree of personalization to the noise in user preferences

We thank you for your answer. We would like to bring important clarifications:

  • First, TPD-uniform is a variant of TPD that replaces, at inference time only, the learned user weights wuw^u by uniform weights wiu=1/nw_i^u = 1/n, while keeping the same basis policies π1,…,πn\pi_1, \dots, \pi_n. These policies are trained jointly with the user weights through the personalized objective of TPD, and thus benefit from personalized weights at training time. The fully non-personalized counterpart is DPO, and Figure 5 (see Table R1) shows a consistent improvement of TPD over DPO for a fixed divergence on three of the four datasets considered in the paper. Note that we already include a capacity-matched variant of DPO, with the same number of trainable parameters as TPD, which rules out that this improvement simply comes from the additional capacity of the nn basis policies.

  • Second, we would like to correct the interpretation, and stress that Table R5 reflects the noisy nature of real preference data, and the ability of TPD to be robust to it rather than its failure in personalizing. On Community Alignment(CA) and PRISM (the two real user datasets), user preferences are noisy and hardly predictable as shown by the low preference accuracies achieved by all personalization baselines implemented in the paper (including LoRe): at most 66% for CA and 62% for PRISM, as opposed to the 75-86% range for synthetic data, and similar to the unpersonalized DPO that already reaches 66% and 61.05%. In this regime, the statistically appropriate per-user estimate moves closer to the population average, and a method that depart strongly from uniform weights would be fitting noise. TPD naturally recovers this behavior, as shown by Table 5 where we observe results close to TPD-uniform.

  • Finally and crucially, we would like to clarify that Table R5 does not contradict the interest of the divergence choice, but rather supports it. It shows that the mass-covering divergences (forward KL and α\alpha-divergence) yield results very close to TPD-uniform, and thus amplify this 'regularizing' effect, making them, as we stressed above, potentially more robust to noise. This is further confirmed by the distribution of the learned user weights (Table R7). On every dataset, forward KL and the α\alpha-divergence learn user weights that are closer to uniform than reverse KL and JSD. Moreover, the weights are much closer to uniform on the real datasets than on PersonalLLM, where the per-user signal is strong and personalized weights clearly improve over uniform ones.

Table R7: Distance of the learned user weights from uniform. Mean over seen users of KL(wu ∥ Un)/log⁡n\mathrm{KL}(w^u \,\|\, U_n)/\log n, where wuw^u are the user's learned weights over the nn adapters, and UnU_n is the uniform distribution. 0 means uniform weights and 1 means all the weight is on a single adapter.

DatasetReverse KLJSDα-div. (α=0.5)Forward KL
Roleplay0.510.530.170.22
PRISM0.300.260.240.24
Community Alignment0.490.470.360.31
PersonalLLM0.790.800.620.66

We will include this discussion and results in the paper. We thank you again for the discussion and we hope that our clarifications and additional experiments alleviate your concerns.

Reviewer_fqXBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes TPD, a framework for personalizing language model generation without user-specific model updates at test time. TPD learns a shared basis of feature-level policies and user-specific preference weights from past pairwise preference data using a generalized DPO objective, without explicitly training reward models. For unseen users, the method estimates preference weights from a small set of comparisons and combines the learned policies through multi-objective decoding. The framework supports different alignment objectives (based on f-divergence) and is evaluated on multiple preference datasets.

Critical strengths

None

The paper provides a concrete implementation of DPO with latent reward features using LoRA adapters and demonstrates personalized decoding for unseen users without updating model parameters at test time. The experiments cover both synthetic and real-world preference datasets, including controlled settings with varying preference diversity and adaptation budgets.

While these are positive aspects, I do not find them sufficient to establish a substantive contribution beyond prior work (or to support acceptance), given the weaknesses.

Critical weaknesses

W1. Insufficient substantive contribution Beyond LoRe

Equations (8) and (9) are essentially identical to equations (18)-(19) in LoRe under reverse KL. LoRe derives a DPO-style objective that jointly learns multiple basis policies and user-specific weights constrained to the simplex, and proposes estimating only the weights for new users while keeping the policies fixed. TPD's formulation differs in per-user loss weighting and adds only \ell_2 regularization during adaptation. Moreover, LoRe explicitly proposes combining learned latent rewards with multi-objective alignment methods to perform personalized generation.

Nevertheless, TPD presents this formulation as a central contribution. The relationship to LoRe is mentioned only in footnote, which describes the LoRe as "for personalized reward learning," without acknowledging that LoRe explicitly derives a policy-learning formulation for language generation.

Thus, TPD's distinguishing contributions appear to be implementing the formulation proposed in LoRe and empirically evaluating it for personalized generation, as well as generalizing it to f-divergences. However, the manuscript presents the underlying formulation as a central contribution, without sufficiently distinguishing what was already in LoRe. Also, the practical benefits of the f-divergence generalization are not sufficiently demonstrated: no divergence consistently outperforms the others in the experiments, and reverse KL achieves the highest average win rate and preference accuracy (table 3, figure 3,). The claimed contributions should more accurately reflect the relationship to prior work.

Compared with equations (18)-(19) in LoRe, what new technical elements are introduced under the reverse KL setting? In my understanding, two differences under reverse KL are: (1) LoRe uses 1/∣Di∣1/|D_i| average and TPD does not; (2) LoRe does not use regularization and TPD does. Do these difference make a benefit? Furthermore, given that reverse KL achieves the best average results, what practical benefits does the f-divergence generalization provide?

W2. Insufficient justification and evaluation against reward-based personalized generation

LoRe, PAL, and PReF can already learn latent reward dimensions, and preference weights for unseen users from preference data. So, it is natural to combine them with existing reward-based generation methods. While the authors emphasize the high cost of a two-stage approach that separately trains policies for each dimension after reward learning, this is required for approaches such as MOD and Rewarded Soups, and is not necessary for reward-guided generation methods such as BoN and value augmented sampling (VAS). In particular, PReF has already combined personalized reward modeling with VAS to perform personalized generation. Nevertheless, the manuscript uses LoRe + BoN as its main reward-based baseline only. This leaves practical necessity and advantages of directly learning a latent policy basis insufficiently demonstrated.

Q2. The two-stage training cost highlighted by the authors does not apply to all reward-based generation methods. Can the authors quantitatively demonstrate the advantages of TPD compared with approaches that combine existing latent reward modeling with reward-based generation? LoRe + BoN alone does not seem sufficient to evaluate reward-based personalized generation. Can the authors compare TPD with existing approaches such as PReF + VAS, or methods that combine the learned rewards from LoRe/PAL with VAS or Rewards-in-Context (https://arxiv.org/abs/2402.10207), to validate its advantages in personalization performance and computational efficiency?

Questions for authors

Please see weaknesses

5 comments
AuthorsOpenReview ↗

W1 (Part1). Insufficient substantive contribution beyond LoRe

Thank you for your careful reading and constructive feedback. We address W1 (Part1) below. The figure and table numbers correspond to those in the revised version of the paper, which we have updated.

W1 (Part1). Insufficient substantive contribution beyond LoRe

We agree that the positioning of our work relative to LoRe could be clarified. In the submitted version, we identified LoRe as an important method for learning personalized reward models (Introduction, Related Work) and as a source of inspiration for our approach (Section 4.1). We also noted in a footnote that the LoRe appendix includes a DPO formulation (Eqs. 18–19) similar to ours in the reverse-KL case. However, we acknowledge that the latter element may have left some ambiguity regarding our contributions beyond LoRe. To clarify this, we detail below the three-fold contribution of our work, highlighting in particular the first contribution, which we believe may have been overlooked in the review.

  1. We establish a novel connection with multi-objective decoding (MOD). As mentioned in your review, LoRe explicitly mentions the potential integration of its approach with multi-objective alignment (MOA) methods. For reward-based methods such as Rewards in Context (RiC), this integration involves providing the weighted sum of rewards learned with LoRe as input. A comparison with such baselines is discussed in our answer to W2. However, while LoRe also explicitly includes MOD as a possible plug-in, please note that a naive integration of LoRe with MOD would require performing preference optimization for each basis reward after learning the reward basis. Our work avoids this by introducing a key idea which is absent from the LoRe paper: leveraging a DPO formulation with latent feature rewards (conceptually already introduced under reverse KL in Equations 18–19 of LoRe), we can directly learn the corresponding policy basis, and express the personalized policy as $\pi^u(y \mid x) = \prod_{i=1}^n \pi_i(y \mid x)^{w_i^u} \exp(\tilde{Z}(x))$ to perform greedy decoding as proposed in MOD: $$y_t \in \operatorname*{argmax} \prod_{i=1}^n \pi_i(y_t \mid y_{<t})^{w_i^u}.$$ Thus, regarding your first question ("Compared with equations (18)–19) in LoRe, what new technical elements are introduced under the reverse KL setting?"): we would like to clarify that, even though we add $\ell_2$ regularization for practical benefits, we do not claim novelty for this formulation in the reverse-KL case. Instead, our contribution in the reverse-KL case lies in leveraging this formulation for efficient test-time decoding via MOD.

  2. To fully establish the connection with MOD and enable the use of diverse alignment objectives, we generalize DPO with latent reward features to a broad class of $f$-divergences, satisfying $0 \notin \operatorname{dom}(\nabla f)$, for which MOD is formulated. Regarding the second question “given that reverse KL achieves the best average results, what practical benefits does the $f$-divergence generalization provide?”, we agree that further demonstrating the practical interest of such generalization in the personalization setting would be valuable, and provide new experiments in the answer to W1 (Part2)

  3. As noted in the strengths of our paper, we implement and empirically validate the complete framework, whereas Equations 18–19 in LoRe are left purely as theoretical formulations. Making this formulation work in practice requires several concrete design choices not developed in LoRe (summarized in Algorithm 3), such as generalizing a DPO library to jointly optimize $n$ LoRA adapters and decoupling learning rates between the shared policy basis and user-specific weights. Furthermore, regarding evaluation, LoRe only measures discriminative preference accuracy on static, held-out pairs. Evaluating personalized generation requires benchmarks simulating user preferences from user description, which we assess using a standard two-sided LLM-as-a-judge setup.

To clarify the positioning in the paper w.r.t. LoRe, we made the following sentence corrections:

[L.89] We show that a generalized formulation of DPO can directly learn a basis of feature-level policies, bypassing the need for explicit reward model training while enabling the personalized combination of logits at test time with multi-objective decoding (MOD). More precisely, to establish this connection with MOD, we build on Bose et al. (2025)’s formulation of latent-feature reward learning in policy space, and extend it to accommodate a broad class of $f$-divergences that can be used in MOD.

[L.268] An essentially equivalent version of this formulation and of the adaptation problem (9) is given in the appendix of \citep{boselore} under reverse KL; here, we derive the corresponding generalized $f$-divergence objective to establish its connection with multi-objective decoding.

AuthorsOpenReview ↗

W1 (Part2). Practical interest of the $f$-divergence generalization

While Reverse KL provides the best results in average in Table 3, we believe that the fact that “no divergence consistently outperforms the others in the experiments” further motivates the generalization to $f$-divergence. To consolidate this, we added the following experiments:

  • We include forward KL in Table 3 (see answer to W2 or updated version of the paper) to enlarge the set of possible $f$-divergences and to fully study the standard spectrum from Reverse KL to Forward KL.
  • We additionally provide in Figure 5 (included in the updated paper and presented in tabular form below with Table R1) a visual comparison of TPD and DPO across the four divergence choices on all datasets. Crucially, while performance decreases from reverse KL to forward KL on PersonalLLM and RP for both DPO and TPD, the opposite trend is observed on PRISM. This suggests that different divergences offer complementary benefits depending on the dataset, as also observed in prior work (Wang et al., 2024a; Go et al., 2023). We hypothesize that the advantage of forward KL (and $\alpha$-divergence) over reverse KL (and JSD) on PRISM stems from its higher level of noise (computed as the proportion of preference labels disagreeing with the user oracle, i.e., LLM judge with user profile). Specifically, reverse KL and JSD induce substantially larger preference margins (i.e., $\log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}$) in DPO (see Figure 10 in Appendix E.2), which potentially amplifies errors on mislabeled pairs.

Table R1: Win rate (%) against the base model per $f$-divergence. Tabular form of Figure 5 in the paper. Bold is the best divergence and $\underline{\text{underlined}}$ is the second best.

DatasetNoise (%)Methodreverse KLJSD$\alpha$-div. ($\alpha=0.5$)forward KL
PersonalLLM ($\gamma=0.01$)0.0TPD72.9±1.4$\underline{70.1\pm1.4}$62.8±1.559.3±1.6
DPO$\underline{\mathit{63.6\pm1.5}}$65.8±1.558.0±1.657.7±1.6
Community Alignment8.8±0.3TPD$\underline{53.8\pm3.0}$53.1±3.151.9±3.155.0±3.0
DPO$\underline{\mathit{54.6\pm3.0}}$58.6±3.151.4±3.352.8±3.3
Roleplay11.6±0.3TPD57.9±3.454.8±3.4$\underline{55.0\pm3.5}$48.4±3.3
DPO55.3±3.5$\underline{\mathit{51.6\pm3.6}}$49.9±4.251.1±4.0
PRISM21.4±0.3TPD47.2±1.845.8±1.850.6±1.8$\underline{50.5\pm1.8}$
DPO45.1±1.944.7±1.8$\underline{\mathit{47.5\pm2.0}}$49.7±2.0
AuthorsOpenReview ↗

W2. Insufficient justification and evaluation against reward-based personalized generation

We agree that the comparison with reward-based personalized generation methods deserved further investigation. Therefore, directly addressing Q2 (“Can the authors compare TPD with existing approaches such as PReF + VAS, or methods that combine the learned rewards from LoRe/PAL with VAS or Rewards-in-Context,to validate its advantages in personalization performance and computational efficiency?”), we extend our comparison beyond LoRe+BoN to include LoRe+RiC and LoRe+VAS across all experiments. As shown in Table 2 of the paper (Table R3 below, restricted to LoRe rows for conciseness), the clear performance gains of our proposed method (TPD) over LoRe+BoN, LoRe+VAS, and LoRe+RiC are consistently confirmed on PersonalLLM.

Table R2: Win rate (%) against the base model on PersonalLLM.

MethodHigh diversity ($\gamma=0.01$) SeenHigh diversity ($\gamma=0.01$) UnseenHigh diversity ($\gamma=0.01$) OverallModerate diversity ($\gamma=0.1$) SeenModerate diversity ($\gamma=0.1$) UnseenModerate diversity ($\gamma=0.1$) Overall
LoRe + BoN67.8 ± 2.161.0 ± 2.264.4 ± 1.562.4 ± 2.257.8 ± 2.260.1 ± 1.5
LoRe + VAS60.8 ± 2.258.8 ± 2.259.8 ± 1.656.6 ± 2.256.8 ± 2.256.7 ± 1.6
LoRe + RiC55.4 ± 2.254.2 ± 2.254.8 ± 1.649.0 ± 2.250.0 ± 2.249.5 ± 1.6
TPD74.6 ± 1.971.2 ± 2.072.9 ± 1.471.4 ± 2.074.4 ± 2.072.9 ± 1.4

We also compared on PersonalLLM the performance–compute trade-offs between TPD and LoRe-based methods by varying key parameters governing compute cost in each method: the number of generations $N$ for BoN, the value-model training dataset size $D$ for VAS, the number of online SFT rounds $r$ for RiC, and the LoRA adapter training steps for TPD (see Table 7 in the Appendix for method details and parameter definitions). The complete results are in Figure 3 (in the updated paper) and summarized in tabular form below in Table R3a-b.

Table R3a: Seen users:best win rate reachable within a compute budget (training + inference wall-clock time).

Time budgetLoRe + BoNLoRe + VASLoRe + RiCTPD
≤ 1 h67.8 ± 2.1–––
≤ 3 h67.8 ± 2.1–48.4 ± 2.265.4 ± 2.1
≤ 6 h67.8 ± 2.153.2 ± 2.252.0 ± 2.269.4 ± 2.1
≤ 12 h67.8 ± 2.154.8 ± 2.255.4 ± 2.273.2 ± 2.0
≤ 24 h67.8 ± 2.159.6 ± 2.255.4 ± 2.273.2 ± 2.0

Table R3b: Unseen users: win rate and computation time (adaptation + inference wall-clock time) for the best configuration on seen

MethodConfigurationTimeWin rate (%)
LoRe + BoN$N=2^7$0.8 h61.0 ± 2.2
LoRe + VAS$D=30$k8.1 h58.8 ± 2.2
LoRe + RiC$r=4$2 min54.2 ± 2.2
TPDsteps = 7k0.7 h71.2 ± 2.0

For seen users, TPD consistently outperforms both the VAS and RiC baselines at comparable computational costs. While BoN incurs lower total compute time because it does not require model training, increasing the compute budget ($N$) yields diminishing returns. For unseen users, TPD achieves substantially higher performance than BoN at comparable costs (a breakdown of training, adaptation, and inference times is provided in Appendix E.1).

We finally included these baselines in Table 3 of the paper (Table R4 below, restricted to LoRe rows for clarity). On Roleplay, TPD maintains a clear advantage over the other methods. However, on the PRISM and CA datasets, LoRe+VAS achieves competitive results with an overall average of $52.0 \pm 1.6%$. In contrast, LoRe+RiC achieves relatively low performance, with an average score of $33.9 \pm 1.4%$.

Table R4: Win rate (%) pooled over seen and unseen users.

MethodRoleplayPRISMCommunity Alignment (CA)Average
LoRe + BoN53.8 ± 3.150.5 ± 1.852.6 ± 3.052.3 ± 1.6
LoRe + VAS50.1 ± 3.452.0 ± 1.853.9 ± 3.052.0 ± 1.6
LoRe + RiC29.6 ± 2.836.3 ± 1.735.8 ± 2.733.9 ± 1.4
TPD (reverse KL)57.9 ± 3.447.2 ± 1.853.8 ± 3.053.0 ± 1.6
TPD (JSD)54.8 ± 3.445.8 ± 1.853.1 ± 3.151.2 ± 1.6
TPD ($\alpha$-div., $\alpha=0.5$)55.0 ± 3.550.6 ± 1.851.9 ± 3.152.5 ± 1.7

Thank you again for your thoughtful feedback and constructive suggestions. We hope that our clarifications and additional experiments alleviate your concerns, and we remain available to answer any remaining questions or follow-ups during the rest of the discussion period

Reviewer_79WgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes TPD, which personalizes generation at test time by modeling each user's reward as a simplex-weighted combination of nn latent reward features that are shared across users. Rather than learning the features and then doing nn separate preference optimizations, TPD inserts the ff-DPO reward-policy correspondence into a LoRe-style joint likelihood and trains nn LoRA policy adapters together with the per-user weights. The weights for a new user are fit by a small regularised logistic regression on the cached per-adapter features, and the responses are then decoded by combining the adapter logits in the style of MOD with rules for reverse KL, JS, and alpha divergences. Experiments with Qwen3-0.6B and Llama-3.2-3B on PersonalLLM, Roleplay, PRISM, and Community Alignment all show that TPD improves the average win rate.

Critical strengths
  • Eq. 8 trains the feature-level policies and user weights in a single ff-DPO-style problem. This eliminates both the explicit reward-feature stage and the nn downstream fine-tuning runs of the naive pipeline. Implementing the basis as LoRA adapters on a frozen shared backbone makes training parameter-efficient. The method is cheap to adapt to new users, requiring no gradient steps.
  • The experiments are informative with good ablations and baselines. The experiments eliminate judge noise. It also includes DPO baselines, a greedy decoding DPO variant, and LoRe on the same LLM embeddings.
Critical weaknesses
  • Very limited methodological novelty. Bose et al. (2025) already present the KL instance of the joint objective. The test-time combination is MOD (Shi et al., 2024), and the reparameterization is ff-DPO (Wang et al., 2024a).
  • Eq. 10 is a sequence-level result, but Algorithm 2 applies it token by token. This is invalid for non-KL divergences, as the gradient of a sequence-level ratio does not decompose over tokens. Even for reverse KL, token-wise greedy decoding only approximates the mode of the sequence-level product distribution and does not sample from it.
Questions for authors
  1. How does TPD with adapted weights compare to TPD with uniform weights?
3 comments
AuthorsOpenReview ↗

W1. Methodological Novelty.

We thank you for your helpful feedback and careful reading. We would like to clarify the novelty of the proposed approach and its importance for LM personalization. The central novelty of our approach is to establish a connection between reward-based multi-objective approaches (in particular, MOD) and reward feature learning approaches (in particular, LoRe). Importantly, to the best of our knowledge, as described in the related work in our paper (see section Multi-objective (MO) Test-time Alignment and Table 1), we are the first to enable multi-objective alignment with unknown objective dimensions and user weights, by learning them from unstructured preference examples (holistic pairwise comparisons). In doing so, our framework enables efficient test-time alignment to realistic human preferences. Furthermore, we realize this connection beyond a naive plug-in of LoRe and MOD, that would induce performing costly preference optimization with each reward features after learning it, by leveraging and extending to f-divergences LoRe’s formulation of DPO with latent reward features.

AuthorsOpenReview ↗

Q1. Comparison with uniform weights

We added a comparison of TPD with adapted weights against TPD with uniform weights (TPD-uniform) across all datasets and divergence choices, in terms of both win rate ratio and preference accuracy. The results are presented in Table R5. On PersonalLLM (synthetic), TPD clearly improves over TPD-uniform across both metrics. On Roleplay, PRISM, and Community Alignment, TPD shows consistent (but minor) improvements over TPD-uniform for Reverse KL and JSD on both metrics, while sometimes underperforming with Forward KL and α-div.

Table R5a: Win rate ratio (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).

DatasetMethodreverse KLJSDα-div. (α=0.5)forward KL
Community AlignmentTPD-uniform53.5±3.052.7±3.056.2±3.057.6±3.0
TPD53.8±3.0 (+0.3)53.1±3.1 (+0.4)51.9±3.1 (-4.3)55.0±3.0 (-2.6)
RoleplayTPD-uniform57.5±3.353.3±3.450.5±3.553.4±3.5
TPD57.9±3.4 (+0.4)54.8±3.4 (+1.5)55.0±3.5 (+4.5)48.4±3.3 (-5.0)
PRISMTPD-uniform47.1±1.844.5±1.751.5±1.950.1±1.8
TPD47.2±1.8 (+0.1)45.8±1.8 (+1.3)50.6±1.8 (-0.9)50.5±1.8 (+0.4)
PersonalLLM (γ=0.01)TPD-uniform65.4±1.565.7±1.558.5±1.655.9±1.6
TPD72.9±1.4 (+7.5)70.1±1.4 (+4.4)62.8±1.5 (+4.3)59.3±1.6 (+3.4)

Table R5b: Preference accuracy (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).

DatasetMethodreverse KLJSDα-div. (α=0.5)forward KL
Community AlignmentTPD-uniform64.8±2.065.0±2.060.7±2.159.3±2.1
TPD66.0±2.0 (+1.2)65.8±2.0 (+0.8)64.4±2.1 (+3.7)62.8±2.1 (+3.5)
RoleplayTPD-uniform73.0±1.772.6±1.771.4±1.766.1±1.8
TPD75.5±1.6 (+2.5)75.1±1.6 (+2.5)72.3±1.7 (+0.9)66.1±1.8 (±0.0)
PRISMTPD-uniform61.5±1.461.1±1.460.7±1.459.5±1.4
TPD62.0±1.4 (+0.5)61.3±1.4 (+0.2)60.5±1.4 (-0.2)59.5±1.4 (±0.0)
PersonalLLM (γ=0.01)TPD-uniform74.5±1.473.1±1.471.5±1.472.4±1.4
TPD86.2±1.1 (+11.7)84.6±1.1 (+11.5)80.8±1.2 (+9.3)77.7±1.3 (+5.3)

This is consistent with the results reported in the paper, showing more modest personalization gains of TPD for simulated personas (Roleplay) and real users (PRISM, Community Alignment), compared to the clear gains observed on PersonalLLM, where users are simulated with deterministic utility functions. Given that TPD remains the best personalization method across all 7 implemented baselines on average in the real user experiments (Table 3 in the paper), we attribute those limited gains to the inherent difficulty of capturing real-user preferences that are scarce, dynamically evolving, and potentially context-dependent. This points to sequential, on-policy adaptation of user weights as an interesting future direction. We have added this discussion to the Conclusion of the revised paper. Table R5a-b are also included in Appendix E.2.3.

AuthorsOpenReview ↗

W2. Greedy Decoding Rule Approximation

We agree that the approximations behind the decoding rule could be further clarified, and we do so below:

  • approximating the mode instead of sampling: using greedy decoding indeed “only approximates the mode of the sequence level product distribution and does not sample from it”, even in the reverse KL case. However, exact sampling from $\pi^u(y\mid x) = \prod_{i=1}^n \pi_i(y\mid x)^{w_i^u}\exp\big(\tilde{Z}(x)\big)$ cannot be done efficiently, as $\tilde{Z}(x)$ involves a sum over all possible responses $y$. In contrast, the mode of $\pi^u$ does not depend on $\tilde{Z}(x)$, which is why we target it and approximate it by token-level greedy decoding as in MOD. To make this clear in the paper, we included the following sentence correction:

[L.307] While exact sampling is intractable due to $\tilde{Z}(x)$, the mode of the policy does not depend on it and can be approximated by the greedy token-level decoding rule: $y_t \in \operatorname*{argmax} \prod_{i=1}^n \pi_i(\cdot \mid y_{<t})^{w_u^i}$.

While the impact of greedy decoding within TPD cannot be assessed directly in the general case of n>1n > 1, as noted in the paper's strengths, our experiments include ``a greedy decoding DPO variant'' alongside its sampled DPO counterpart (i.e., TPD(n=1)\text{TPD}(n=1)), showing close performance (sampled: 66.9, greedy: 67.5; see Table~2 in the paper).

  • token-level objective instead of sequence-level: we agree that the "sequence-level ratio does not decompose over tokens", and added in the revised paper (Appendix A, L.868) a clarification, as well as an empirical illustration, which we summarize below.

The token-level rule of Algorithm 2 approximates the maximum of the token-level objective Stok(y)=∑t=1∣y∣[log⁡πref(yt∣y<t,x)+log⁡Fw(ρ1(yt),…,ρn(yt))]S_{\mathrm{tok}}(y) = \sum_{t=1}^{|y|} \big[\log \pi_{\mathrm{ref}}(y_t\mid y_{<t},x) + \log F_w\big(\rho_1(y_t), \ldots, \rho_n(y_t)\big)\big], whereas the mode of the sequence-level distribution maximizes the objective Sseq(y)=log⁡πref(y∣x)+log⁡Fw(ρ1(y),…,ρn(y))S_{\mathrm{seq}}(y) = \log \pi_{\mathrm{ref}}(y\mid x) + \log F_w\big(\rho_1(y), \ldots, \rho_n(y)\big), where Fw(ρ1,…,ρn)=∇f−1(∑i=1nwiu∇f(ρi))F_w(\rho_1, \ldots, \rho_n) = \nabla f^{-1}\big(\sum_{i=1}^n w_i^u \nabla f(\rho_i)\big) denote an aggregation function defined by ff, and ρi(y)=πi(y∣x)πref(y∣x)=∏t=1∣y∣ρi(yt)\rho_i(y) = \frac{\pi_i(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} = \prod_{t=1}^{|y|} \rho_i(y_t), with ρi(yt)=πi(yt∣y<t,x)πref(yt∣y<t,x)\rho_i(y_t) = \frac{\pi_i(y_t\mid y_{<t},x)}{\pi_{\mathrm{ref}}(y_t\mid y_{<t},x)}.

For reverse KL, ∇f(ρ)=log⁡ρ+1\nabla f(\rho) = \log \rho + 1, so that FwF_w reduces to the weighted geometric mean ∏i=1nρiwiu\prod_{i=1}^n \rho_i^{w_i^u}, which commutes with products, so that Stok=SseqS_{\mathrm{tok}} = S_{\mathrm{seq}} and the token-level rule targets exactly the mode of the sequence-level distribution. However, this is not necessarily the case for other ff-divergences. For instance, for forward KL, ∇f(ρ)=−1/ρ\nabla f(\rho) = -1/\rho and FwF_w is the weighted harmonic mean (∑i=1nwiuρi−1)−1\big(\sum_{i=1}^n w_i^u \rho_i^{-1}\big)^{-1}, which does not commute with products, so that Stok≠SseqS_{\mathrm{tok}} \neq S_{\mathrm{seq}} in general.

We illustrate the impact of this approximation in Table R6 (see Figure 6 in the paper), where we score up to 10 real candidate responses to the same prompt for 100 users per dataset with both objectives, and report the share of users for whom both objectives select the same best response, and the Kendall τ\tau between the two rankings of the candidates. As expected, reverse KL yields perfect agreement on both metrics. Agreement varies across the other divergences, JSD being the loosest approximation, but the same best response is selected for more than 90% of users in all cases, with τ≥0.94\tau \geq 0.94. This comparison remains limited, as it relies on small sets of candidate responses.

Table R6: Agreement between the token-level objective StokS_{\mathrm{tok}} and the sequence-level objective SseqS_{\mathrm{seq}}. Share of users (%) for whom both objectives select the same best response / Kendall τ\tau between the two rankings of the candidates, averaged over 100 users per dataset.

DivergencePersonalLLMRoleplayCommunity AlignmentPRISM
Reverse KL100 / 1.000100 / 1.000100 / 1.000100 / 1.000
JSD93 / 0.94192 / 0.98396 / 0.98898 / 0.992
α\alpha-div. (α\alpha=0.5)100 / 0.986100 / 0.999100 / 1.000100 / 0.998
Forward KL99 / 0.98699 / 0.992100 / 0.99898 / 0.997

We also included details on the MOD approximation $\tilde{Z}(x)=0$ in Appendix A.

Thank you again for your thoughtful feedback and constructive suggestions. We hope that our clarifications and additional experiments alleviate your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.