acceptodds
Under review as a conference paper at ICLR 2027

Fast & efficient online alignment with multidimensional reward model

Abstract

We consider online alignment of language model outputs to user-specific preferences. We assume each user's preferences are driven by an unknown function over a chosen multi-dimensional vector of measurable text qualities (e.g., helpfulness, conciseness, factuality), so that personalisation reduces to learning that function from live feedback. Online adaptation without updating the underlying LLM requires learning this function from sparse and heterogeneous feedback, with no guarantee that any structural assumption about how qualities combine fits every user. Under a KL-regularised objective, a standard identity gives the optimal policy as an exponentially tilted base distribution whose tilt is the user's reward; with it assumed to be a function of qualities, learning the user's preferences therefore reduces to learning that reward from feedback. We cast this as a bandit problem and solve it with Thompson sampling under two posterior families: a linear- Gaussian belief and a non-parametric Gaussian process (GP). In synthetic experiments, the linear variant converges within tens of interactions where its assumption holds, and the GP variant succeeds on non-linear structures where the linear model degrades. On both the synthetic worlds and a PRISM-derived persona benchmark, achieved reward depends jointly on what the posterior has learned and on how sharply the decoder acts on it. We characterise this dependence and show its role alongside the choice of quality basis in the gap to an in-context baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language model alignment, online learning, linear bandits, Thompson sampling, KL regularisation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RvkeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the important problem of online alignment of language models to user-specific preferences. The authors assume that user preferences can be represented as an unknown function over a multidimensional vector of pretrained reward scores, such as helpfulness, harmlessness, and humor. Under this assumption, the problem essentially reduces to classical function approximation from noisy feedback. The authors use Thompson sampling with either a linear-λ\lambda Gaussian belief or a nonparametric GP to capture potentially nonlinear interactions between preference dimensions. They also use sampling-importance resampling (SIR) or argmax for response selection under a KL-regularized objective.

Critical strengths
  1. Online and sample-efficient personalization: Unlike approaches that require policy fine-tuning or training additional models, this framework learns user preferences online from limited feedback while keeping the underlying LLM and reward models frozen.

  2. Multiple preference dimensions: The method supports reward representations with 3–19 preference dimensions, including potentially nonlinear interactions, without relying on the LLM's in-context learning capabilities.

Critical weaknesses
  1. Limited novelty: Although the setting is interesting, the underlying problem reduces to estimating a noisy function over fixed reward features. The proposed linear-Gaussian model, GP, and Thompson sampling are well-established techniques in regression and bandit literature. While their combination is useful, I find it difficult to identify a substantial new methodological insight or a distinctive challenge specific to LLMs that the paper addresses.

  2. Necessity of SIR during learning: While SIR is motivated for inference-time response selection, is reward-based candidate selection necessary during preference-function approximation? The posterior can also be updated using randomly selected or more informative samples. Given that the authors already generate a large candidate pool, could selecting a few diverse candidates or using posterior uncertainty to identify uncertain rankings improve sample efficiency? A comparison with simpler feedback-acquisition strategies would strengthen the paper.

  3. Computational efficiency and candidate limitations: Since the underlying policy remains frozen, the quality of alignment is limited by the diversity and quality of generated candidates. Furthermore, using K=100K=100 responses per prompt introduces substantial inference-time overhead. For example, 10,000 prompts would require approximately one million candidate generations, in addition to reward evaluations. Can the authors provide end-to-end computational costs and compare them with methods that incur an initial training cost but have cheaper inference? It is unclear whether this approach is practical for large-scale interactive agents.

  4. Limited benefit of online personalization: In Appendix A.15, the online linear model achieves performance comparable to a frozen model fitted using only five feedback observations, while a generic reward head without personalization performs better than both. This raises questions about the practical benefit of continuous online learning and whether simpler regression-based approaches could achieve similar results.

Questions for authors
  1. Novelty and effectiveness: Given that the proposed methods rely on established regression and bandit techniques, what is the main methodological contribution? Can the authors demonstrate a clear advantage of online Bayesian learning over simpler approaches, such as ridge regression or a frozen preference estimator?

  2. Feedback acquisition: Is reward-based candidate selection necessary during preference learning? Could random sampling, uncertainty-based selection, or selecting a small set of diverse candidates provide more informative feedback and achieve similar performance with fewer interactions?

  3. Computational efficiency: Given that the method requires generating and evaluating K=100K=100 candidates per prompt, what is the end-to-end computational cost at scale? How does this compare with approaches that incur an initial training cost but offer cheaper inference, especially for applications involving thousands of prompts?

4 comments

W4/Q1: "[...] the online linear model achieves performance comparable to a frozen model fitted using only five feedback observations, while a generic reward head without personalization performs better than both." "Can the authors demonstrate a clear advantage of online Bayesian learning over [...] ridge regression or a frozen preference estimator?"

We agree (Section 5.4; Appendix A.15, Table 6), and trace the PRISM result to two properties of the quality bases.

When personas agree, and no quality basis sees how they differ. They share most of their rating variation (Appendix A.17, Table 8), and per-user weights explain almost none of the user-specific part on any basis (Table 9), so no per-user learner can do much better than a shared score.

ArmoRM 19 heads are strongly correlated, so after 100 ratings most posterior directions remain uncertain and a draw reorders the candidates about as much as the learned weights do. Serving the posterior mean of the same posterior beats the n=5n=5 fit and comes within one point of the generic head:

PRISM, ArmoRM-19, persona rating (0–100)online, Thompson draw (paper)online, posterior mean
served reward60.964.7
minus a frozen five-label ArmoRM-19 fit (60.9)+0.05±1.29+0.05 \pm 1.29 (4/10)+3.86±0.91+3.86 \pm 0.91 (9/10)
minus the generic head (65.6)−4.68±1.12-4.68 \pm 1.12 (0/10)−0.88±0.34-0.88 \pm 0.34 (2/10)

Paired per user, mean ± SE over 10 users (users ahead); one five-label draw per user, as in Table 6 (Appendix A.22, Table 17C). On HHH, whose heads explain less of the ratings (Table 9), the posterior mean ties the five-label fit (−0.55±0.35-0.55 \pm 0.35).

The new results below show where online learning adds value.

1. When users disagree, one agent per user pays, on the qualities the basis scores. We ran 6 P-SOUPS personas whose preferences conflict in pairs (Jang et al., 2023: elementary vs. PhD-level answers, concise vs. informative, friendly-humorous vs. unfriendly), rated by the same Opus persona scorer, with the paper's PRISM agent unchanged, on 100 new prompts with replies of up to 1,024 tokens. The table shows the persona's score of the served replies (0 to 100), averaged over the 100 rounds (Appendix A.25, Table 21).

six conflicting personas, ArmoRM-19persona score of served replies (0–100)one agent per persona minus this arm
one agent per persona21.96
one agent for all six personas20.85+1.10±0.18+1.10 \pm 0.18
generic head (ArmoRM's overall score, no learning)19.22+2.74±0.40+2.74 \pm 0.40

Mean over 20 seeds and 6 personas, mean ± SE over the 100 prompts.

  • The lead over the shared agent grows with rounds, from +0.4+0.4 in rounds 1 to 50 to +1.8+1.8 in rounds 51 to 100.
  • It comes from the qualities ArmoRM-19 scores (verbosity, complexity): the informative, concise and elementary personas gain. The PhD-level and the two tone personas do not: ArmoRM-19 has no tone head, and few of Qwen's candidates suit them.
  • Where the basis sees little of how users differ, a shared agent wins: on 1,000 PersonalLLM users, where ArmoRM-19 sees only 18% of the user-specific reward, one agent per user trails one shared agent (noise-free ratings, paper protocol, 50 rounds).
  • Adding the generic head as one more quality score nearly matches it where users agree and beats it where they differ (synthetic worlds in which the generic head sees what users share; Appendix A.25, Table 22).

2. Online learning is worth 5 to 25 offline labels, and almost never falls below a random pick. In the synthetic worlds of Figures 1 and 2, a frozen fit is a ridge regression (or a GP) on nn rated replies, fitted once. Each run gives one capture value over its 100 rounds; a run is one random set of rated replies for a frozen fit (200 per profile) or one seed for the online agent (10 per profile for linear TS, 200 for GP TS), and mean, median and worst run are over all runs of all profiles.

linear worlds, captureσ\sigmameanmedianworst runruns with capture < 0 (below a random pick)
frozen ridge, n=50.250.950.99−1.35-1.350.9%
frozen ridge, n=250.251.001.000.960%
online linear TS (paper)0.250.970.970.920%
frozen ridge, n=51.00.840.94−1.65-1.652.5%
frozen ridge, n=251.00.980.990.740%
online linear TS (paper)1.00.930.940.830%
non-linear worlds, captureσ\sigmameanmedianworst runruns with capture < 0 (below a random pick)
frozen ridge, n=250.250.570.56−0.81-0.812.1%
frozen GP, n=50.250.500.56−0.44-0.445.0%
frozen GP, n=250.250.870.900.250%
online GP TS (paper)0.250.860.880.480%
frozen ridge, n=251.00.550.53−0.81-0.812.8%
frozen GP, n=51.00.280.32−0.69-0.6921%
frozen GP, n=251.00.630.68−0.22-0.220.8%
online GP TS (paper)1.00.650.69−0.12-0.120.1%
  • Which labels a fit gets matters: some fits do worse than a random pick (last column), against 1 online run in 2,140.
  • In linear worlds online learning ties a five-label fit at low noise, beats it under noise, and loses to a 25-label fit (Appendix A.21), although by the last 25 rounds it serves as well. Its value is that it needs no labels before deployment and recovers from a bad start.
  • When the basis sees only part of the user's reward, online learning's lead over a five-label fit grows, while a 25-label fit stays slightly ahead (Appendix A.21, Table 15B).
  • On PRISM, too, which five labels a fit gets matters (Table 6 has one draw per user). Scored by each user's own direction in the basis (a regression on the user's ~1,000 ratings, which tracks the persona's ratings), about a fifth of five-label draws do worse than a random pick and Table 6's HHH draw is a lucky one, while the online agent's final posterior mean never falls below a random pick.

3. When the fit's labels come from the same user, online learning leads in every setting. The tables above give a frozen fit its labels for free. If the user must first rate nn random replies, those rounds earn no capture in expectation and the fit serves the remaining 100−n100 - n rounds (same fits as above: ridge in linear worlds, GP in non-linear ones).

capture over 100 roundsσ\sigmaonline (paper)rate 5, then serve the fitrate 25, then serve the fit
linear worlds0.250.970.900.75
linear worlds1.00.930.800.73
non-linear worlds0.250.860.480.65
non-linear worlds1.00.650.260.48

(Appendix A.21, Table 16B)

W2/Q2: "is reward-based candidate selection necessary during preference-function approximation?" "Could random sampling, uncertainty-based selection, or selecting a small set of diverse candidates provide more informative feedback [...]?"

In the paper the user rates the reply we serve, which is the best pick of the learned reward. The user could rate another candidate instead, for example the most uncertain one or a random one, but in our tests this makes little difference. On PRISM we show the information the belief gains about the user's weights (nats of posterior precision over 100 rounds) and the persona rating (0 to 100) relative to rating the served reply.

PRISM, posterior-mean argmax served; user ratesinformation, HHH (nats)information, ArmoRM-19 (nats)reward vs. served, HHHreward vs. served, ArmoRM-19
the served reply12.744.600
the most uncertain candidate14.348.7−1.05±0.49-1.05 \pm 0.49−0.03±0.49-0.03 \pm 0.49
a random candidate13.344.9−2.07±0.87-2.07 \pm 0.87−1.06±0.39-1.06 \pm 0.39

On the synthetic worlds we show capture, averaged over rounds.

synthetic, σ=1\sigma = 1, capture; user ratesTS, linear worldsTS, non-linear worldsupdate-and-argmax, linear worldsupdate-and-argmax, non-linear worlds
the served reply0.9380.5350.9590.458
the most uncertain candidate0.9570.5180.9710.530
a random candidate0.9410.4760.9630.503

PRISM: paired per user, mean ± SE over 10 users. Synthetic: 20 runs per profile.

  • On PRISM every rule gathers about the same information, and none gains reward over rating the served reply.
  • On the synthetic worlds the rules are close under either serving rule: under update-and-argmax, rating the most uncertain candidate helps a little on linear worlds, and its larger gains on non-linear worlds are within noise.
  • Rating a small set per round (the served reply plus 1 or 3 diverse candidates) saves one or two rounds on the linear worlds but needs more ratings in total, and diverse, uncertain and random sets perform similarly.
  • Each round's fresh pool already makes the served reply informative and explores when serving the posterior mean (reply to 6rX6, W2); the other rules also ask the user to rate a reply they did not receive.

W3/Q3: "[...] the quality of alignment is limited by the diversity and quality of generated candidates. [...] what is the end-to-end computational cost at scale?"

End-to-end cost is linear in the number of prompts, and 10,000 prompts take a few H100-hours. Each prompt is one round, and each round generates its KK candidates in one batched request that shares the prompt (per-round cost in point 2 of our comment to all reviewers, and in the revised manuscript, Appendix A.18, Tables 11 and 12). At K=100K = 100, 10,000 prompts therefore mean one million candidates but only 10,000 batched requests. One frozen LLM and one set of scorers serve all users, and each user adds only a state of DD weights and a D×DD \times D covariance (for linear model). The table shows H100-hours for 10,000 prompts and the lead over ICL in persona rating, as in point 2:

10,000 promptsH100-hourslead over ICL, HHHlead over ICL, ArmoRM-19
ours, K=100K = 1004 to 7+4.7±0.9+4.7 \pm 0.9+7.2±1.4+7.2 \pm 1.4
ours, K=10K = 101.7 to 2.0+3.8±0.5+3.8 \pm 0.5+5.6±1.2+5.6 \pm 1.2
ICLabout 1.500
  • We agree that the candidates bound the reward (Section 6, Appendix A.19, Table 13), but in our tests they do not slow learning.

  • Per-user fine-tuning would break even after C/0.09C / 0.09 s rounds per user, where CC is its training cost. A fine-tuned model generates one reply per round (0.51 s; Appendix A.18, Table 11), against K=10K = 10 replies (0.6 s; Table 12), so a fine-tune that costs one GPU-minute pays for itself only after about 670 rounds per user (60 s / 0.09 s). Additionally, it needs training data and a model per user before deployment; we did not run per-user fine-tuning.

W1/Q1: "I find it difficult to identify a substantial new methodological insight or a distinctive challenge specific to LLMs [...]" "what is the main methodological contribution?"

The challenge specific to LLMs is the decoder, and our methodological finding is that its sharpness confounds comparisons. The agent acts by tilting the LLM's distribution towards the learned reward: sharpening pays only along a learned direction (Section 5.3, Figure 4), and decoders that spend similar KL against the base LLM reach the same capture (Appendix A.20, Table 14). Personalisation methods should therefore be compared at matched KL; the paper's τ=1\tau = 1 tilt spends almost none on PRISM (Figure 5). Our contribution is not a new bandit algorithm but a plug-and-play formulation of interaction-time personalisation of a frozen LLM, and an empirical account of when each of its parts pays. The decoder is one of the four limits it identifies (see point 1 of our comment to all reviewers).

In the revised manuscript (new material in blue): frozen fits versus online learning in Appendix A.21 (Tables 15, 16); which reply the user rates in A.23 (Tables 18, 19); cost in A.18 (Tables 11, 12); nested pools in A.19 (Table 13); decoders at matched KL in A.20 (Table 14); Thompson sampling versus the posterior mean in A.22 (Table 17); conflicting personas and the generic head as a quality score in A.25 (Tables 21, 22).

Reviewer_yYnROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the online personalization of language models. Without updating the weights of the language model, the authors frame the problem as a bandit problem where the reward is a function of a set of qualities of the model's response. The authors then investigated two methods for solving the Bayesian bandit problem, namely a linear Gaussian belief and a Gaussian process, from different types of signals including scalar, binary, and pairwise feedbacks. Through experiments on synthetic preferences and a PRISM-derived persona benchmark, the authors compare the Bayesian bandit methods to an in-context learning baseline and characterize how achieved reward depends jointly on posterior belief accuracy and decoding sharpness.

Critical strengths

This paper frames the online alignment problem as an inference problem, providing a new angle to personalization. Under this formulation, the parameters of the underlying language model stay frozen at interaction time and thus personalization is computationally cheap.

Critical weaknesses

Under the proposed framework, the core problem is inference and is independent of language models. In fact, both the linear Gaussian model and the Gaussian process are standard machine learning models that are not specific to language models. The synthetic experiments can also be carried out without language models. Thus this paper provides little new knowledge to the community.

The computational costs of the investigated algorithms are not discussed. The baseline algorithm is an in-context learning method where one response is sampled at each round. In contrast, the SIR method employed by the proposed algorithms need to sample multiple responses at each round, incurring much higher computational costs. This is very important as the serving cost of modern language models is significant. However, the authors fail to address this important matter in the paper.

Questions for authors
  1. Can the authors clarify the take-away messages from this paper and how they are connected to the language model community?

  2. Can the authors discuss the computational cost of the Thompson Sampling approach?

1 comment

W2: "The computational costs of the investigated algorithms are not discussed. [...] the SIR method employed by the proposed algorithms need to sample multiple responses at each round, incurring much higher computational costs." Q2: "Can the authors discuss the computational cost of the Thompson Sampling approach?"

Thompson sampling itself adds only the belief step (draw, select, update), which takes about 0.1 ms per round for the linear model. Generating and scoring the candidates is the decoding step of any method that samples then selects (e.g. MAP, Soft Best-of-n), and at K=10K = 10 a round costs about as much as an ICL round (point 2 of our comment to all reviewers). The revised manuscript adds this cost analysis in Appendix A.18 (Tables 11 and 12).

W1: "The synthetic experiments can also be carried out without language models."

The posterior update is standard Bayesian regression, but the experiments cannot run without an LLM: the candidates the agent can serve are generated at inference time by the LLM. Each round's candidates are Qwen2.5-7B-Instruct completions to an HH-RLHF prompt, scored by real reward models; only each synthetic user's reward over these scores is synthetic. The policy is a tilt of these samples, so the LLM's pool caps the reward (Appendix A.19, Table 13) and how far the decoder moves from the LLM decides how much of what is learned reaches the user (Appendix A.20, Table 14).

W1: "the core problem is inference and is independent of language models. [...] Thus this paper provides little new knowledge to the community." Q1: "Can the authors clarify the take-away messages from this paper and how they are connected to the language model community?"

The problem is not independent of LMs, because the agent can only act through the LLM. Each round its options are KK samples from the LLM and its policy is a tilt of the LLM's distribution, so the reward is capped by what the LLM generates and by how far the decoder moves from it. The ingredients are standard. The take-aways are the four findings in point 1 of our comment to all reviewers: what the quality basis, the inference, the uncertainty and the decoder each limit.

Reviewer_6rX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates online personalization as Bayesian inference over fixed response-quality features, using linear or Gaussian-process reward models and Thompson sampling to select responses from a frozen LLM. Experiments evaluate the approach on synthetic reward functions and PRISM-derived simulated users.

Critical strengths
  • The method is easy to follow. It clearly separates scoring the answers, learning the user’s preferences, and choosing which answer to show. This also makes it easier to understand the role of each component.

  • The comparison between linear and GP models is useful. It shows when a simple linear model works and when a more flexible model can help. The experiments also examine how the answer-selection rule affects the results.

Critical weaknesses
  • The main assumption needs more support. The method assumes that a user’s preferences can be predicted from the fixed quality scores. However, the same user may want humor in a casual message but not in a formal one. Although the scorers take the prompt as input, the paper does not show whether their scores keep enough information to capture these differences. This is a critical problem because in many cases only part of the attributes matter in a specific task.

  • The added value of Bayesian online learning is not clear. Once the quality scores are fixed, most of the learning components are standard. The paper needs stronger evidence that tracking uncertainty and using Thompson sampling lead to better decisions than simply updating a preference model and choosing its highest-scoring answer. The comparison with ICL alone does not clearly show this.

  • The experiments leave a gap to real use. Since the method mainly builds on established ideas, I would put more weight on whether it offers a clear practical benefit in a realistic setting. The evaluation assumes feedback at every round and uses simulated users. It also consists of separate question–answer rounds, so it is unclear how the method would extend to an ongoing multi-turn conversation, where each response affects what the user says next. In a real application, users may only give feedback occasionally, or mainly when they are unhappy. It remains unclear how much effort users would need to spend before seeing a benefit, and whether that benefit would justify the extra cost of generating and scoring multiple answers.

Questions for authors
  1. What evidence shows that the fixed quality scores can capture changes in what the same user wants across different tasks? How can we tell whether the scores retain enough information about the situation?

  2. What real application is this setup intended for? In practice, what kinds of user preferences can it adapt to, and how much is this limited by the choice of scoring dimensions and the accuracy of the scorers? How would the method work in a real multi-turn conversation, where each answer affects the user’s next response and feedback may only be available occasionally? Would the resulting benefits justify the user effort and the cost of generating and scoring multiple answers?

2 comments

W2: "The paper needs stronger evidence that tracking uncertainty and using Thompson sampling lead to better decisions than simply updating a preference model and choosing its highest-scoring answer."

With scalar feedback, tracking uncertainty paid only where exploration was needed: on the non-linear worlds, where Thompson sampling ends the run with lower regret on all five but leads only slightly on average, and in a control that offers the same 100 candidates in every round, as in a classic bandit. We ran a baseline, which uses the same Bayesian update but serves the argmax of the posterior mean instead of the argmax of a posterior draw. We show capture averaged over rounds, cumulative reward at T=100T = 100, and the served reply's pseudo-regret (best candidate minus served reply) averaged over the last 10 rounds.

synthetic, argmaxserving rulecapturecumulative rewardpseudo-regret, last 10 rounds
GP, non-linear worlds, σ=0.5\sigma = 0.5Thompson sampling (posterior draw)0.773346.70.075
posterior mean0.727337.60.128
GP, non-linear worlds, σ=1\sigma = 1Thompson sampling (posterior draw)0.646327.30.209
posterior mean0.584315.80.397
linear, non-linear worlds, σ=1\sigma = 1Thompson sampling (posterior draw)0.535305.50.571
posterior mean0.458283.90.860
linear, linear worlds, σ=1\sigma = 1Thompson sampling (posterior draw)0.938501.40.016
posterior mean0.959505.50.011

On PRISM we show reward, the persona's feedback of the chosen reply (0 to 100).

PRISM, linear, argmaxThompson sampling (posterior draw)posterior meandifference (users where TS is ahead)
HHH58.459.2−0.75±0.57-0.75 \pm 0.57 (4/10)
ArmoRM-1960.964.7−3.8±1.0-3.8 \pm 1.0 (0/10)
  • The reviewer's doubt is justified: each round's fresh pool already explores, so on most worlds TS and serving the posterior mean do the same job. Only in the classic-bandit control, which serves the same 100 candidates every round, does using the posterior mean learn less than TS.
  • On non-linear worlds TS still pays (Appendix A.22, Table 17A): serving the posterior mean can lock onto worse replies, overrating what it serves and ranking the best reply far down, while a TS draw keeps trying other candidates.
  • On ArmoRM-19, the draw adds noise (Appendix A.22, Table 17B and C): its correlated heads leave most posterior directions near the prior, and a draw's random weights on them reorder the candidates: late in the run it serves the posterior mean's top reply in a quarter of rounds, against half on HHH.

W3/Q2: "The evaluation assumes feedback at every round and uses simulated users. [...] users may only give feedback occasionally, or mainly when they are unhappy. [...]"

With occasional ratings the linear belief still converges, but unhappy-only ratings break the GP. We replayed the synthetic runs (σ=0.25\sigma = 0.25) with the user rating a random 1 in 10 rounds, or only rounds whose reward is in the bottom quarter of what they have received so far ("unhappy"; Appendix A.24, Table 20; reward and pseudo-regret curves in Figure 8). We show capture averaged over all 100 rounds and over the last 25, and, as in Figures 1 and 2, cumulative reward at T=100T = 100 and pseudo-regret averaged over the last 10 rounds.

user model, worldsuser ratesratings in 100 roundscapture, all roundscapture, last 25 roundscumulative rewardpseudo-regret, last 10 rounds
linear, linearevery round1000.971.00506.90.001
linear, lineara random 1 in 10 rounds120.780.99464.10.007
linear, linearunhappy rounds only210.900.98487.00.053
GP, non-linearevery round1000.850.96356.60.015
GP, non-lineara random 1 in 10 rounds110.510.72297.00.105
GP, non-linearunhappy rounds only230.260.30234.30.410
  • The linear belief learns more slowly with fewer ratings but converges either way. For the GP, a likelihood that models the censoring is the natural fix.

Other practical considerations:

  • Scoring dimensions and scorer accuracy. The quality basis caps what the method can learn, and the learner reaches most of that cap. How much a basis explains a user's feedback can be checked before deployment on a pilot set of rated replies, as Table 9 does with PRISM ratings (Appendix A.17), and for each user only from their own ratings once deployed. With conflicting personas the agent learned length and complexity, which ArmoRM-19 scores, but not tone (Appendix A.25, Table 21; reply to Rvke, W4). In synthetic worlds, scores that miss part of the reward or come from noisy scorers lower the cap, and online TS still reaches most of it (Appendix A.21, Table 15B and C).
  • Cost. At K=10K = 10 a round costs about as much as an ICL round and keeps most of the lead over ICL (point 2 of our comment to all reviewers; cost analysis in the revised manuscript, Appendix A.18).
  • Simulated users. Each persona is Claude Opus 4.5 conditioned on one real PRISM evaluator's survey answers and rated conversations, and it predicts held-out ratings better than the mean baseline (Appendix A.8, Table 1); a human study remains future work.
  • Multi-turn. We have not tested it. The natural way to do so is to treat each turn as a round: generate and score candidates given the conversation so far, and keep the user's belief across turns and conversations; this is future work.

Comments on the paper

2 comments

We thank the reviewers for their reading and comments. We address two concerns that appear in all three reviews here, and the replies to each reviewer cover the rest. Any new results use the paper's setup (Section 4) and are in the revised manuscript, marked in blue (Appendices A.18 to A.25). In the following comments, W and Q refer to the weaknesses and questions of each review, numbered in the order they appear.

For synthetic worlds, the ground-truth reward r⋆r^\star for each synthetic user is known, so we also report the share of the achievable reward gain that the agent captures: capture=1T∑t=1Tr⋆(yt)−rˉt⋆max⁡kr⋆(yt,k)−rˉt⋆,rˉt⋆=1K∑k=1Kr⋆(yt,k),\text{capture} = \frac{1}{T} \sum_{t=1}^{T} \frac{r^\star(y_t) - \bar r^\star_t}{\max_k r^\star(y_{t,k}) - \bar r^\star_t}, \qquad \bar r^\star_t = \frac{1}{K} \sum_{k=1}^{K} r^\star(y_{t,k}), with candidates yt,1,…,yt,Ky_{t,1}, \dots, y_{t,K} and served reply yty_t in round tt. Capture is 0 for a random pick, 1 for the best candidate every round, and negative below a random pick.

1. What does the paper contribute? (yYnR W1/Q1, Rvke W1/Q1, 6rX6 W2)

Our contribution is not a new bandit algorithm but a plug-and-play formulation of interaction-time personalisation of a frozen LLM, and an empirical account of when each of its parts pays. Its ingredients (Thompson sampling, linear and GP reward models, the KL-regularised tilt with SIR) are standard, and the paper says so (Section 1, Contributions).

  • The framework. Any quality basis, user model (linear-λ\boldsymbol\lambda or GP) and feedback type (scalar, binary; pairwise for linear) plugs into one belief loop, with no training of the LLM or the scorers. Each round the frozen LLM generates candidates, the scorers place them on the basis q\boldsymbol q, the belief selects one through the tilt, and the user's feedback updates it (Section 3, Algorithm 1). It leads ICL under every feedback type and on every PRISM basis (Appendix A.12, Table 4; Appendix A.14, Table 5).
  • Why the LLM setting matters. The LLM limits the reward through both its candidates and the decoder. Each round the agent can only pick from a fresh pool of LLM samples, which caps the reward (Appendix A.19, Table 13), and it acts only through how sharply it tilts the LLM's distribution.

Our central finding is that representation, inference and decoding impose distinct limits, and improving one does not lift another.

  • Representation. The basis, not the learner, sets what per-user learning can gain. Under argmax, the basis moves the lead over ICL as much as the decoder (Section 5.4), and with conflicting personas one agent per persona beats a shared agent and the generic head (Appendix A.25, Table 21). On PRISM the personas largely agree and no basis sees how they differ, so a shared score is the better choice (Appendix A.17, Tables 8 and 9; reply to Rvke, W4).
  • Inference. Online learning recovers the user's reward without labels before deployment. Each belief learns the reward it is built for, the GP where the linear model levels off (Section 5.1, Figures 1 and 2). Online learning almost never falls below a random pick and beats every frozen fit whose labels the same user must rate first (Appendix A.21, Tables 15 and 16).
  • Uncertainty. Thompson sampling pays only where exploration is needed. It wins on non-linear worlds (Appendix A.22, Table 17) and in a classic-bandit control that reuses one pool. Elsewhere each fresh pool already explores, so serving the posterior mean ties or wins (reply to 6rX6, W2).
  • Decoding. How sharply the decoder acts decides how much of what is learned reaches the user, so methods should be compared at matched KL. Argmax beats the paper's default tilt along a learned direction but not along a random one (Section 5.3, Figure 4). Decoders that spend similar KL perform alike, and the default tilt spends almost none on PRISM (Appendix A.20, Table 14; Figure 5).

2. What does the method cost? (yYnR W2/Q2, Rvke W3/Q3, 6rX6 W3/Q2)

Generating 100 candidates takes about twice as long as generating one, not 100 times, because they share one prompt. Each round generates KK candidates, scores them, and runs the belief (draw, select, update); the LLM is not trained. The table gives each step's complexity and its measured time per round. This cost analysis is new in the revised manuscript (Appendix A.18, Tables 11 and 12, referenced from Section 4).

per-round stepcomplexitymeasured at K=100K = 100
generation of KK candidates (one batch, shared prompt)prefill O(L)O(L), paid once; decoding O(Km)O(Km) tokens1.04 s (vs. 0.51 s at K=1K = 1 )
scoring of KK candidates by DD scorer modelsO(K)O(K) calls per scorer; scorers can run concurrently, bounded by the slowest one0.39 s (ArmoRM-19, one model); 0.71 s (HHH, three models in parallel, one GPU each; vs. 1.45 s in sequence)
linear belief: update, draw, selection over KKupdate O(D2)O(D^2), draw O(D3)O(D^3), selection O(KD)O(KD); independent of tt0.1 ms
GP belief: update, draw, selection over KKupdate O(t2)O(t^2), draw and selection O(Kt+t2)O(Kt + t^2)about 45 ms at t=100t = 100

LL: prompt tokens. mm: tokens per reply. DD: number of quality scores. tt: rounds so far. We measured each on one H100 (vLLM, Qwen2.5-7B-Instruct, as in Appendix A.9).

In vLLM the candidates share the prompt and its KV cache, so a round costs Tprefill(L)+Tdecode(K)T_{\text{prefill}}(L) + T_{\text{decode}}(K) (encode the prompt once, decode KK replies in one batch) instead of K (Tprefill(L)+Tdecode(1))K\,(T_{\text{prefill}}(L) + T_{\text{decode}}(1)). Decoded tokens still grow with KK, so a GPU already busy with many users serves fewer of them.

Alternatives pay elsewhere. ICL's prompt grows with every stored example, and per-user fine-tuning needs a training run and a model per user (break-even in the reply to Rvke, W3).

KK trades cost for reward: ten candidates keep most of the lead over ICL at less than half the time, and an ICL round takes about 0.5 s. Lead over ICL: paired difference in persona rating (0 to 100, mean over 100 rounds), mean ± SE over 10 users.

quality basisKKtime per roundlead over ICL
HHH100.6 s+3.8±0.5+3.8 \pm 0.5
HHH1001.8 s+4.7±0.9+4.7 \pm 0.9
ArmoRM-19100.6 s+5.6±1.2+5.6 \pm 1.2
ArmoRM-191001.4 s+7.2±1.4+7.2 \pm 1.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.