- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the important problem of online alignment of language models to user-specific preferences. The authors assume that user preferences can be represented as an unknown function over a multidimensional vector of pretrained reward scores, such as helpfulness, harmlessness, and humor. Under this assumption, the problem essentially reduces to classical function approximation from noisy feedback. The authors use Thompson sampling with either a linear- Gaussian belief or a nonparametric GP to capture potentially nonlinear interactions between preference dimensions. They also use sampling-importance resampling (SIR) or argmax for response selection under a KL-regularized objective.
Critical strengths
-
Online and sample-efficient personalization: Unlike approaches that require policy fine-tuning or training additional models, this framework learns user preferences online from limited feedback while keeping the underlying LLM and reward models frozen.
-
Multiple preference dimensions: The method supports reward representations with 3–19 preference dimensions, including potentially nonlinear interactions, without relying on the LLM's in-context learning capabilities.
Critical weaknesses
-
Limited novelty: Although the setting is interesting, the underlying problem reduces to estimating a noisy function over fixed reward features. The proposed linear-Gaussian model, GP, and Thompson sampling are well-established techniques in regression and bandit literature. While their combination is useful, I find it difficult to identify a substantial new methodological insight or a distinctive challenge specific to LLMs that the paper addresses.
-
Necessity of SIR during learning: While SIR is motivated for inference-time response selection, is reward-based candidate selection necessary during preference-function approximation? The posterior can also be updated using randomly selected or more informative samples. Given that the authors already generate a large candidate pool, could selecting a few diverse candidates or using posterior uncertainty to identify uncertain rankings improve sample efficiency? A comparison with simpler feedback-acquisition strategies would strengthen the paper.
-
Computational efficiency and candidate limitations: Since the underlying policy remains frozen, the quality of alignment is limited by the diversity and quality of generated candidates. Furthermore, using responses per prompt introduces substantial inference-time overhead. For example, 10,000 prompts would require approximately one million candidate generations, in addition to reward evaluations. Can the authors provide end-to-end computational costs and compare them with methods that incur an initial training cost but have cheaper inference? It is unclear whether this approach is practical for large-scale interactive agents.
-
Limited benefit of online personalization: In Appendix A.15, the online linear model achieves performance comparable to a frozen model fitted using only five feedback observations, while a generic reward head without personalization performs better than both. This raises questions about the practical benefit of continuous online learning and whether simpler regression-based approaches could achieve similar results.
Questions for authors
-
Novelty and effectiveness: Given that the proposed methods rely on established regression and bandit techniques, what is the main methodological contribution? Can the authors demonstrate a clear advantage of online Bayesian learning over simpler approaches, such as ridge regression or a frozen preference estimator?
-
Feedback acquisition: Is reward-based candidate selection necessary during preference learning? Could random sampling, uncertainty-based selection, or selecting a small set of diverse candidates provide more informative feedback and achieve similar performance with fewer interactions?
-
Computational efficiency: Given that the method requires generating and evaluating candidates per prompt, what is the end-to-end computational cost at scale? How does this compare with approaches that incur an initial training cost but offer cheaper inference, especially for applications involving thousands of prompts?
4 comments
2. Online learning is worth 5 to 25 offline labels, and almost never falls below a random pick. In the synthetic worlds of Figures 1 and 2, a frozen fit is a ridge regression (or a GP) on rated replies, fitted once. Each run gives one capture value over its 100 rounds; a run is one random set of rated replies for a frozen fit (200 per profile) or one seed for the online agent (10 per profile for linear TS, 200 for GP TS), and mean, median and worst run are over all runs of all profiles.
| linear worlds, capture | mean | median | worst run | runs with capture < 0 (below a random pick) | |
|---|---|---|---|---|---|
| frozen ridge, n=5 | 0.25 | 0.95 | 0.99 | 0.9% | |
| frozen ridge, n=25 | 0.25 | 1.00 | 1.00 | 0.96 | 0% |
| online linear TS (paper) | 0.25 | 0.97 | 0.97 | 0.92 | 0% |
| frozen ridge, n=5 | 1.0 | 0.84 | 0.94 | 2.5% | |
| frozen ridge, n=25 | 1.0 | 0.98 | 0.99 | 0.74 | 0% |
| online linear TS (paper) | 1.0 | 0.93 | 0.94 | 0.83 | 0% |
| non-linear worlds, capture | mean | median | worst run | runs with capture < 0 (below a random pick) | |
|---|---|---|---|---|---|
| frozen ridge, n=25 | 0.25 | 0.57 | 0.56 | 2.1% | |
| frozen GP, n=5 | 0.25 | 0.50 | 0.56 | 5.0% | |
| frozen GP, n=25 | 0.25 | 0.87 | 0.90 | 0.25 | 0% |
| online GP TS (paper) | 0.25 | 0.86 | 0.88 | 0.48 | 0% |
| frozen ridge, n=25 | 1.0 | 0.55 | 0.53 | 2.8% | |
| frozen GP, n=5 | 1.0 | 0.28 | 0.32 | 21% | |
| frozen GP, n=25 | 1.0 | 0.63 | 0.68 | 0.8% | |
| online GP TS (paper) | 1.0 | 0.65 | 0.69 | 0.1% |
- Which labels a fit gets matters: some fits do worse than a random pick (last column), against 1 online run in 2,140.
- In linear worlds online learning ties a five-label fit at low noise, beats it under noise, and loses to a 25-label fit (Appendix A.21), although by the last 25 rounds it serves as well. Its value is that it needs no labels before deployment and recovers from a bad start.
- When the basis sees only part of the user's reward, online learning's lead over a five-label fit grows, while a 25-label fit stays slightly ahead (Appendix A.21, Table 15B).
- On PRISM, too, which five labels a fit gets matters (Table 6 has one draw per user). Scored by each user's own direction in the basis (a regression on the user's ~1,000 ratings, which tracks the persona's ratings), about a fifth of five-label draws do worse than a random pick and Table 6's HHH draw is a lucky one, while the online agent's final posterior mean never falls below a random pick.
3. When the fit's labels come from the same user, online learning leads in every setting. The tables above give a frozen fit its labels for free. If the user must first rate random replies, those rounds earn no capture in expectation and the fit serves the remaining rounds (same fits as above: ridge in linear worlds, GP in non-linear ones).
| capture over 100 rounds | online (paper) | rate 5, then serve the fit | rate 25, then serve the fit | |
|---|---|---|---|---|
| linear worlds | 0.25 | 0.97 | 0.90 | 0.75 |
| linear worlds | 1.0 | 0.93 | 0.80 | 0.73 |
| non-linear worlds | 0.25 | 0.86 | 0.48 | 0.65 |
| non-linear worlds | 1.0 | 0.65 | 0.26 | 0.48 |
(Appendix A.21, Table 16B)
W2/Q2: "is reward-based candidate selection necessary during preference-function approximation?" "Could random sampling, uncertainty-based selection, or selecting a small set of diverse candidates provide more informative feedback [...]?"
In the paper the user rates the reply we serve, which is the best pick of the learned reward. The user could rate another candidate instead, for example the most uncertain one or a random one, but in our tests this makes little difference. On PRISM we show the information the belief gains about the user's weights (nats of posterior precision over 100 rounds) and the persona rating (0 to 100) relative to rating the served reply.
| PRISM, posterior-mean argmax served; user rates | information, HHH (nats) | information, ArmoRM-19 (nats) | reward vs. served, HHH | reward vs. served, ArmoRM-19 |
|---|---|---|---|---|
| the served reply | 12.7 | 44.6 | 0 | 0 |
| the most uncertain candidate | 14.3 | 48.7 | ||
| a random candidate | 13.3 | 44.9 |
On the synthetic worlds we show capture, averaged over rounds.
| synthetic, , capture; user rates | TS, linear worlds | TS, non-linear worlds | update-and-argmax, linear worlds | update-and-argmax, non-linear worlds |
|---|---|---|---|---|
| the served reply | 0.938 | 0.535 | 0.959 | 0.458 |
| the most uncertain candidate | 0.957 | 0.518 | 0.971 | 0.530 |
| a random candidate | 0.941 | 0.476 | 0.963 | 0.503 |
PRISM: paired per user, mean ± SE over 10 users. Synthetic: 20 runs per profile.
- On PRISM every rule gathers about the same information, and none gains reward over rating the served reply.
- On the synthetic worlds the rules are close under either serving rule: under update-and-argmax, rating the most uncertain candidate helps a little on linear worlds, and its larger gains on non-linear worlds are within noise.
- Rating a small set per round (the served reply plus 1 or 3 diverse candidates) saves one or two rounds on the linear worlds but needs more ratings in total, and diverse, uncertain and random sets perform similarly.
- Each round's fresh pool already makes the served reply informative and explores when serving the posterior mean (reply to 6rX6, W2); the other rules also ask the user to rate a reply they did not receive.
W3/Q3: "[...] the quality of alignment is limited by the diversity and quality of generated candidates. [...] what is the end-to-end computational cost at scale?"
End-to-end cost is linear in the number of prompts, and 10,000 prompts take a few H100-hours. Each prompt is one round, and each round generates its candidates in one batched request that shares the prompt (per-round cost in point 2 of our comment to all reviewers, and in the revised manuscript, Appendix A.18, Tables 11 and 12). At , 10,000 prompts therefore mean one million candidates but only 10,000 batched requests. One frozen LLM and one set of scorers serve all users, and each user adds only a state of weights and a covariance (for linear model). The table shows H100-hours for 10,000 prompts and the lead over ICL in persona rating, as in point 2:
| 10,000 prompts | H100-hours | lead over ICL, HHH | lead over ICL, ArmoRM-19 |
|---|---|---|---|
| ours, | 4 to 7 | ||
| ours, | 1.7 to 2.0 | ||
| ICL | about 1.5 | 0 | 0 |
-
We agree that the candidates bound the reward (Section 6, Appendix A.19, Table 13), but in our tests they do not slow learning.
-
Per-user fine-tuning would break even after s rounds per user, where is its training cost. A fine-tuned model generates one reply per round (0.51 s; Appendix A.18, Table 11), against replies (0.6 s; Table 12), so a fine-tune that costs one GPU-minute pays for itself only after about 670 rounds per user (60 s / 0.09 s). Additionally, it needs training data and a model per user before deployment; we did not run per-user fine-tuning.
W1/Q1: "I find it difficult to identify a substantial new methodological insight or a distinctive challenge specific to LLMs [...]" "what is the main methodological contribution?"
The challenge specific to LLMs is the decoder, and our methodological finding is that its sharpness confounds comparisons. The agent acts by tilting the LLM's distribution towards the learned reward: sharpening pays only along a learned direction (Section 5.3, Figure 4), and decoders that spend similar KL against the base LLM reach the same capture (Appendix A.20, Table 14). Personalisation methods should therefore be compared at matched KL; the paper's tilt spends almost none on PRISM (Figure 5). Our contribution is not a new bandit algorithm but a plug-and-play formulation of interaction-time personalisation of a frozen LLM, and an empirical account of when each of its parts pays. The decoder is one of the four limits it identifies (see point 1 of our comment to all reviewers).
In the revised manuscript (new material in blue): frozen fits versus online learning in Appendix A.21 (Tables 15, 16); which reply the user rates in A.23 (Tables 18, 19); cost in A.18 (Tables 11, 12); nested pools in A.19 (Table 13); decoders at matched KL in A.20 (Table 14); Thompson sampling versus the posterior mean in A.22 (Table 17); conflicting personas and the generic head as a quality score in A.25 (Tables 21, 22).
We agree (Section 5.4; Appendix A.15, Table 6), and trace the PRISM result to two properties of the quality bases.
When personas agree, and no quality basis sees how they differ. They share most of their rating variation (Appendix A.17, Table 8), and per-user weights explain almost none of the user-specific part on any basis (Table 9), so no per-user learner can do much better than a shared score.
ArmoRM 19 heads are strongly correlated, so after 100 ratings most posterior directions remain uncertain and a draw reorders the candidates about as much as the learned weights do. Serving the posterior mean of the same posterior beats the n=5 fit and comes within one point of the generic head:
Paired per user, mean ± SE over 10 users (users ahead); one five-label draw per user, as in Table 6 (Appendix A.22, Table 17C). On HHH, whose heads explain less of the ratings (Table 9), the posterior mean ties the five-label fit (−0.55±0.35).
The new results below show where online learning adds value.
1. When users disagree, one agent per user pays, on the qualities the basis scores. We ran 6 P-SOUPS personas whose preferences conflict in pairs (Jang et al., 2023: elementary vs. PhD-level answers, concise vs. informative, friendly-humorous vs. unfriendly), rated by the same Opus persona scorer, with the paper's PRISM agent unchanged, on 100 new prompts with replies of up to 1,024 tokens. The table shows the persona's score of the served replies (0 to 100), averaged over the 100 rounds (Appendix A.25, Table 21).
Mean over 20 seeds and 6 personas, mean ± SE over the 100 prompts.