acceptodds
Under review as a conference paper at ICLR 2027

Online LLM Personalization from Pairwise Preferences Across Users

Abstract

Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 41–46% of random pairs' excess regret and about three fifths of what a per-user linear oracle fitted in hindsight removes. We release code, prompts, and the benchmark.

Primary area:
reinforcement learning
Keywords:
contextual dueling bandits, dataset and benchmark, pairwise preference feedback, prompt selection, personalized alignment, preference learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_5p17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper frames the problem of online personalization as a contextual dueling bandit problem. The authors propose a a method that learns a gaussian mixture prior from previous users and then use bayesian logistic regression to fit new users.

Critical strengths
  1. The online llm personalization is an important problem. The author created a good abstraction that allows them to establish sound mathematical models behind it
Critical weaknesses
  1. Personalization settings are way too toy. The paper only has 6 persona, and the reward model is a linear utility model. I found this setting to be too vanilla to be representitive as realistic personalization setting.
  2. There is little novelty in personalization algorithm design. It seems that the author simply moved gaussian mixture priors and bayesian logistic regression into the field of llm personalization, where the setting itself is already highly simplified.
  3. The authors assume that user preference vary across questions, but from the experiments it's unclear whether the model directly learned the user's preferred persona than being question conditioned.
Questions for authors

See weakness

Reviewer_2ufbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for tutoring preferences with 6 synthetic teaching styles and pairwise judgements from LLM simulated users. They also introduce a baseline algorithm for adaptation, which learns a mixed prior over users which is only used if the prior's choices are better than predicting the user's choices in isolation. They study the efficacy of their baseline algorithm on a numerical simulator and the tutoring preference setting.

Critical strengths
  • the proposed baseline method has been shown to be effective against other bandit algorithms on the 3 new synthetic students in terms of hard regret over the run.
Critical weaknesses
  • The scale of the benchmark is quite small and synthetic with limited users/questions that are all synthetically judged with LLMs.
  • The proposed method has lower utility than fitting a linear bandit to each user, which questions the efficacy of the prior that is learnt.
  • there is an assumption of a linear utility model which limits the realism of the tasks in which this can be applied to.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.