acceptodds
Withdrawn from ICLR 2027

Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning from human feedback, preference alignment, pareto learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2qcoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an MPG-DPO method to improve sample efficiency in personalized LLM alignment. It constructs a personalized preference cone from historical comparisons and multidimensional evaluations, then uses score confidence bounds to filter confidently dominated responses and select pairs with strong unresolved trade-offs. Experiments across multiple LLM families show improved performance under matched preference-query budgets.

Critical strengths
  1. The paper addresses a practically important problem. Improving sample efficiency in online preference alignment can reduce the burden of costly preference feedback, particularly for personalized alignment.

  2. MPG-DPO combines a personalized preference cone, confidence-aware filtering, and trade-off selection to incorporate historical user comparisons into response pair selection.

  3. MPG-DPO achieves consistent improvements across multiple LLM families under matched preference-query budgets, and the component ablations support the contribution of its key components.

Critical weaknesses
  1. Table 1 reports performance under a single fixed preference-query budget. Although MPG-DPO consistently improves the reported mean performance, the modest gains do not establish whether its advantage persists across different budgets or how many queries it saves to reach comparable performance.

  2. Table 4 evaluates adaptation to different preference oracles, but does not clearly distinguish general improvements in response quality from gains attributable to learning user-specific trade-offs from historical comparisons.

  3. Why does MPG-DPO use a ridge-regression estimator to predict candidate scores rather than directly applying Prometheus2 to all candidate responses before response pair selection? Comparing these alternatives would clarify the predictor’s practical value. If such a comparison cannot be provided, the paper should explain why the predictor is preferable to direct scoring.

  4. The writing of the introduction could be further improved. The motivation is developed through a lengthy chain of arguments, making the central narrative difficult to follow. The transitions to the proposed approach are not sufficiently clear, and the method overview contains repetition without clearly explaining how the components address the stated problem.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_91jrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies sample selection for online preference alignment. MPG-DPO constructs and updates a personalized preference cone from historical pairwise comparisons and multidimensional response scores. It uses ridge-regression predictions with confidence bounds to filter confidently dominated candidates, then selects a pair exhibiting unresolved trade-offs through a max–min criterion. The resulting comparison is used for a standard online DPO update. Experiments cover three 7B model families and four evaluation tasks, with LLMs providing preference feedback, and report improvements under matched preference-query budgets.

Critical strengths

The paper addresses a relevant problem: allocating expensive preference queries using information about user preferences beyond model-centric uncertainty. The filtering and trade-off selection mechanisms are interpretable, and the component ablations help assess their contributions. Table 3 provides useful controls by matching candidate-pool size, scoring and preference-query budgets, predictor architecture, and warm-up data. The reported mean improvements are consistent across the three evaluated policy families.

Critical weaknesses
  1. The accounting of total preference queries and the practical cost–benefit trade-off need clarification. The matched-budget comparisons support improved utilization of online preference queries. However, the paper does not clearly report the total number of preference-oracle calls, including those used to initialize the preference cone, or explain whether initialization queries are included in the reported budgets. Appendix B shows that warm-up data affect performance, while describing initialization as offline preprocessing does not resolve how its annotation cost is accounted for. Although Table 3 shares warm-up data across methods, the corresponding accounting for the main comparisons in Table 1 remains unclear. Reporting the initialization and online query counts separately, together with performance against cumulative query count, would make the sample-efficiency claim more directly assessable. A further limitation concerns practical cost: Table 3 controls several resource inputs, but end-to-end runtime, generation and scoring costs, and the overhead of cone construction and pair selection are not quantified. These measurements would clarify when the observed quality gains justify the required resources; their absence limits conclusions about overall cost efficiency rather than invalidating the matched-query results.
  2. The experiments do not sufficiently isolate user-specific personalization from improvements in broadly shared response quality. Table 4 replaces the preference oracle with three different LLMs and evaluates each trained policy using its corresponding oracle. This is a reasonable test of adaptation to a target evaluator, but different model identities do not by themselves establish distinct user preference profiles. The paper does not report disagreement on common response pairs or demonstrate contrasting trade-offs between conflicting dimensions. Consequently, the results do not distinguish adaptation to user-specific preferences from improvements favored by all three evaluators. The experimental controls also need clarification. Section 4.1 mentions a fixed judging template, but the supplied paper does not provide the complete judging prompts or specify how preference settings are controlled across Table 4. Moreover, the multidimensional scoring rubrics are generated by the preference oracle; it is unclear whether these rubrics remain fixed when the oracle changes. If they are regenerated, changes in preference labels and scoring criteria could both contribute to the results. Stronger evidence would come from explicitly conflicting preference profiles, controlled prompts and rubrics, and comparisons between matched and cross-user-mismatched histories under otherwise identical conditions. Evaluation with real users or user-identified human preference data would further strengthen external validity.
Questions for authors

Please see the Weaknesses above.

Reviewer_zqHTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MPG-DPO, a sample-efficient online preference alignment method that leverages multidimensional evaluations and historical user feedback to construct personalized preference cones. It filters out dominated responses and selects informative pairs with unresolved preference trade-offs. Theoretical analysis and experiments demonstrate its effectiveness in reducing wasted queries and improving alignment performance over existing baselines.

Critical strengths
  1. The authors make a commendable effort to study online preference alignment within a theoretical framework and provide rigorous analysis of their proposed method.
  2. The paper provides theoretical guarantees showing that the fraction of queries involving dominated responses vanishes asymptotically under certain assumptions.
Critical weaknesses
  1. Insufficient motivation for the preference-cone formulation. While the preference-cone formulation is an interesting approach to modeling user preferences, the paper does not provide sufficient motivation or justification for why this formulation is appropriate for LLM preference alignment. The authors could better explain the intuition behind this formulation and its advantages over alternative approaches to modeling user preferences.
  2. Limited justification of theoretical assumptions. The main theoretical results rely on Assumptions 1 and 2, but the paper provides limited discussion of their validity in practical LLM alignment scenarios. It would strengthen the theoretical contribution if the authors could further justify these assumptions or discuss how restrictive they are in practice.
  3. Limited analysis of computational efficiency. MPD-DPO introduces additional computational overhead through candidate generation, score prediction, and preference-cone construction. The paper lacks runtime or computational cost comparisons with existing methods.
Questions for authors
  1. Could the authors empirically validate the regret bound and convergence behavior established in Theorem 1, perhaps through controlled synthetic or tabular experiments?
  2. Could the authors evaluate some examples of responses generated by LLMs aligned using MPG-DPO? Such examples would help illustrate the practical benefits of the proposed method and provide a more intuitive understanding of its effectiveness beyond quantitative benchmark results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.