- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TPD, which jointly learns a shared policy basis and user-specific weights from preference data. For new users, it fits only the weights and combines the policies during decoding, without further model fine-tuning. The method extends to several f-divergence objectives and is evaluated on synthetic and human preference datasets, with stronger gains in the synthetic setting.
Critical strengths
-
Adapting only a small weight vector while keeping the policy basis frozen is a practical way to personalize generation without maintaining a separate model for each user.
-
The controlled synthetic experiments show promising gains on unseen users with just eight preference examples. This supports the approach when user preferences follow the assumed shared latent structure.
Critical weaknesses
-
LoRe already derives joint policy-basis learning and adaptation of new users’ weights, while MOD provides the decoding framework. The contribution beyond combining these methods seems limited, and the experiments do not show a consistent benefit from the added f-divergence choices.
-
The real user results do not establish a clear personalization benefit. TPD averages 53.0% versus 52.3% for LoRe+BoN, with mixed results on unseen users and only one training seed. A comparison against uniform or shuffled user weights would help isolate the value of personalization.
Questions for authors
See weakness
4 comments
W2. Clear personalization benefit on real users
We agree that further isolating the contribution of personalization on real users would be valuable. To this end, we added a comparison of TPD with adapted weights against TPD with uniform weights (TPD-uniform) across all datasets and divergence choices, in terms of both win rate ratio and preference accuracy. The results are presented in Table R5. On PersonalLLM (synthetic), TPD clearly improves over TPD-uniform across both metrics. On Roleplay, PRISM, and Community Alignment, TPD shows consistent (but minor) improvements over TPD-uniform for Reverse KL and JSD on both metrics, while sometimes underperforming with Forward KL and α-div.
Table R5a: Win rate ratio (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).
| Dataset | Method | reverse KL | JSD | α-div. (α=0.5) | forward KL |
|---|---|---|---|---|---|
| Community Alignment | TPD-uniform | 53.5±3.0 | 52.7±3.0 | 56.2±3.0 | 57.6±3.0 |
| TPD | 53.8±3.0 (+0.3) | 53.1±3.1 (+0.4) | 51.9±3.1 (-4.3) | 55.0±3.0 (-2.6) | |
| Roleplay | TPD-uniform | 57.5±3.3 | 53.3±3.4 | 50.5±3.5 | 53.4±3.5 |
| TPD | 57.9±3.4 (+0.4) | 54.8±3.4 (+1.5) | 55.0±3.5 (+4.5) | 48.4±3.3 (-5.0) | |
| PRISM | TPD-uniform | 47.1±1.8 | 44.5±1.7 | 51.5±1.9 | 50.1±1.8 |
| TPD | 47.2±1.8 (+0.1) | 45.8±1.8 (+1.3) | 50.6±1.8 (-0.9) | 50.5±1.8 (+0.4) | |
| PersonalLLM (γ=0.01) | TPD-uniform | 65.4±1.5 | 65.7±1.5 | 58.5±1.6 | 55.9±1.6 |
| TPD | 72.9±1.4 (+7.5) | 70.1±1.4 (+4.4) | 62.8±1.5 (+4.3) | 59.3±1.6 (+3.4) |
Table R5b: Preference accuracy (%). Mean ± standard error, pooled over seen and unseen users. In parentheses we indicate the gain of TPD over TPD-uniform (positive differences in bold).
| Dataset | Method | reverse KL | JSD | α-div. (α=0.5) | forward KL |
|---|---|---|---|---|---|
| Community Alignment | TPD-uniform | 64.8±2.0 | 65.0±2.0 | 60.7±2.1 | 59.3±2.1 |
| TPD | 66.0±2.0 (+1.2) | 65.8±2.0 (+0.8) | 64.4±2.1 (+3.7) | 62.8±2.1 (+3.5) | |
| Roleplay | TPD-uniform | 73.0±1.7 | 72.6±1.7 | 71.4±1.7 | 66.1±1.8 |
| TPD | 75.5±1.6 (+2.5) | 75.1±1.6 (+2.5) | 72.3±1.7 (+0.9) | 66.1±1.8 (±0.0) | |
| PRISM | TPD-uniform | 61.5±1.4 | 61.1±1.4 | 60.7±1.4 | 59.5±1.4 |
| TPD | 62.0±1.4 (+0.5) | 61.3±1.4 (+0.2) | 60.5±1.4 (-0.2) | 59.5±1.4 (±0.0) | |
| PersonalLLM (γ=0.01) | TPD-uniform | 74.5±1.4 | 73.1±1.4 | 71.5±1.4 | 72.4±1.4 |
| TPD | 86.2±1.1 (+11.7) | 84.6±1.1 (+11.5) | 80.8±1.2 (+9.3) | 77.7±1.3 (+5.3) |
This is consistent with the results reported in the paper, showing more modest personalization gains of TPD for simulated personas (Roleplay) and real users (PRISM, Community Alignment), compared to the clear gains observed on PersonalLLM, where users are simulated with deterministic utility functions. Given that TPD remains the best personalization method across all 7 implemented baselines on average in the real user experiments (Table 3 in the paper), we attribute those limited gains to the inherent difficulty of capturing real-user preferences that are scarce, dynamically evolving, and potentially context-dependent. This points to sequential, on-policy adaptation of user weights as an interesting future direction. We have added this discussion to the Conclusion of the revised paper. Table R5a-b are also included in Appendix E.2.3.
Thank you again for your feedback and constructive suggestions. We hope that our clarifications and additional experiments alleviate your concerns, and we remain available to answer any remaining questions or follow-ups during the rest of the discussion period
Thank you for running the additional experiments and for clarifying how TPD avoids a separate second stage of policy optimization compared with a naive combination of LoRe and MOD.
However, the new results in Table R5 reinforce my main concern about personalization on real users. On both real user datasets, adapting user weights gives almost no win rate gain over TPD with uniform weights under reverse KL and JSD, and even hurts performance under alpha divergence and forward KL. In fact, on both Community Alignment and PRISM, the highest win rate in Table R5a is mostly achieved by uniform weights, and standard DPO reaches 58.6% on Community Alignment in Table R1.
This also affects the interpretation of f divergences in W1. Although forward KL and alpha divergence improve overall win rates on noisier datasets in Table R1, Table R5a shows that personalized weights mostly match or underperform uniform weights under those divergences, indicating that the gain comes from regularizing the shared basis.
Overall, while the method works well on synthetic preferences, the empirical evidence for personalization on real human preferences remains weak, so I will maintain my current score.
TPD adapts its degree of personalization to the noise in user preferences
We thank you for your answer. We would like to bring important clarifications:
-
First, TPD-uniform is a variant of TPD that replaces, at inference time only, the learned user weights by uniform weights , while keeping the same basis policies . These policies are trained jointly with the user weights through the personalized objective of TPD, and thus benefit from personalized weights at training time. The fully non-personalized counterpart is DPO, and Figure 5 (see Table R1) shows a consistent improvement of TPD over DPO for a fixed divergence on three of the four datasets considered in the paper. Note that we already include a capacity-matched variant of DPO, with the same number of trainable parameters as TPD, which rules out that this improvement simply comes from the additional capacity of the basis policies.
-
Second, we would like to correct the interpretation, and stress that Table R5 reflects the noisy nature of real preference data, and the ability of TPD to be robust to it rather than its failure in personalizing. On Community Alignment(CA) and PRISM (the two real user datasets), user preferences are noisy and hardly predictable as shown by the low preference accuracies achieved by all personalization baselines implemented in the paper (including LoRe): at most 66% for CA and 62% for PRISM, as opposed to the 75-86% range for synthetic data, and similar to the unpersonalized DPO that already reaches 66% and 61.05%. In this regime, the statistically appropriate per-user estimate moves closer to the population average, and a method that depart strongly from uniform weights would be fitting noise. TPD naturally recovers this behavior, as shown by Table 5 where we observe results close to TPD-uniform.
-
Finally and crucially, we would like to clarify that Table R5 does not contradict the interest of the divergence choice, but rather supports it. It shows that the mass-covering divergences (forward KL and -divergence) yield results very close to TPD-uniform, and thus amplify this 'regularizing' effect, making them, as we stressed above, potentially more robust to noise. This is further confirmed by the distribution of the learned user weights (Table R7). On every dataset, forward KL and the -divergence learn user weights that are closer to uniform than reverse KL and JSD. Moreover, the weights are much closer to uniform on the real datasets than on PersonalLLM, where the per-user signal is strong and personalized weights clearly improve over uniform ones.
Table R7: Distance of the learned user weights from uniform. Mean over seen users of , where are the user's learned weights over the adapters, and is the uniform distribution. 0 means uniform weights and 1 means all the weight is on a single adapter.
| Dataset | Reverse KL | JSD | α-div. (α=0.5) | Forward KL |
|---|---|---|---|---|
| Roleplay | 0.51 | 0.53 | 0.17 | 0.22 |
| PRISM | 0.30 | 0.26 | 0.24 | 0.24 |
| Community Alignment | 0.49 | 0.47 | 0.36 | 0.31 |
| PersonalLLM | 0.79 | 0.80 | 0.62 | 0.66 |
We will include this discussion and results in the paper. We thank you again for the discussion and we hope that our clarifications and additional experiments alleviate your concerns.
W1. Methodological Novelty and Benefit from f-divergence choices.
We thank you for your helpful feedback and careful reading. We would like to first clarify the novelty of the proposed approach and its importance for LM personalization. The central novelty of our approach is to establish a connection between reward-based multi-objective approaches (in particular, MOD) and reward feature learning approaches (in particular, LoRe). Importantly, to the best of our knowledge, as described in the related work in our paper (see section Multi-objective (MO) Test-time Alignment and Table 1), we are the first to enable multi-objective alignment with unknown objective dimensions and user weights, by learning them from unstructured preference examples (holistic pairwise comparisons). In doing so, our framework enables efficient test-time alignment to realistic human preferences. Furthermore, we realize this connection beyond a naive plug-in of LoRe and MOD, that would induce performing costly preference optimization with each reward features after learning it, by leveraging and extending to f-divergences LoRe’s formulation of DPO with latent reward features.
We also agree that further illustrating the interest of this extension to f-divergence could be valuable. Therefore, we made the following addition in the revised paper:
Table R1: Win rate (%) against the base model per $f$-divergence. Tabular form of Figure 5 in the paper. Bold is the best divergence and $\underline{\text{underlined}}$ is the second best.