acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

Abstract

Reinforcement Learning from Human Feedback (RLHF) can violate fundamental social-choice axioms, including Pareto optimality, majority consistency, pairwise-majority consistency, and Condorcet consistency. Prior work relates these failures to the aggregation rule implicitly implemented by reward learning: unrestricted Bradley–Terry training recovers the Borda count, which satisfies Pareto optimality but not the other axioms. Restricting the reward model does not generally resolve the problem; linear and any fixed-degree polynomial reward models can fail even Pareto optimality. This appears at odds with the empirical success of RLHF and motivates us to ask when its axiomatic guarantees can be recovered. We first show that, with unrestricted rewards, replacing comparison frequencies by majority labels changes the induced rule from Borda to Copeland, thereby recovering all four axioms. For linear rewards, we characterize the regularized estimator as a projection of its unrestricted counterpart and identify how feature geometry and regularization determine whether these guarantees survive. For polynomial rewards, we establish a sharp distinction in the order of quantifiers: although every fixed degree admits an adverse candidate set, every fixed finite candidate set has an interpolation degree beyond which unrestricted aggregation is recovered. Above this threshold, standard RLHF recovers Borda and hence Pareto optimality, whereas majority-based RLHF recovers Copeland and satisfies all four axioms. Together, these results show how aggregation design and reward expressivity jointly determine the axiomatic behavior of RLHF. Experiments on UltraFeedback and Chatbot Arena confirm this division of labor: the gap between the two label choices opens only as a reward model's capacity approaches the per-prompt unrestricted fit, and practical reward models remain far from that regime.

Primary area:
learning theory
Keywords:
RLHF, Social Choice Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yY9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work builds on Axioms for AI Alignment from Human Feedback (Ge et al., NeurIPS 2024). At a high level, it examines how the choice of preference aggregation method and the expressivity of the reward model jointly determine whether reward learning satisfies key social-choice axioms.

Critical strengths

An important and well-motivated problem. How alignment methods reconcile conflicting human judgments is an important question in its own right. I think the problem(s) the paper addresses are important.

An insightful connection to Copeland. The Copeland result is insightful and interesting in a way. The underlying voting rules are well known, but the point is still useful for anyone designing alignment methods.

Critical weaknesses

Relationship to established results. Ge et al. (2024) already studied majority-based loss functions. Once the unrestricted majority-label solution is identified with Copeland, the axiom guarantees follow directly from known properties of Copeland. The paper should make these connections more explict relative to prior work and their stated contributions.

The distinction between aggregation and expressivity has also been discussed before. For example, Section 3 of the survey by Halpern et al (2026) (AI Alignment From Social Choice Perspectives) identifies the aggregation rule that unrestricted BT implements, and then explains how restricting the reward class leads to further failures, including violations of unanimity. The paper should not present this distinction as a new conceptual contribution.

Scope of the guarantees. The theory assumes a fixed candidate set, balanced comparisons, and regularization on candidate scores. The experiments use models shared across prompts, but the main guarantees say nothing about how these models behave on unseen prompts or responses, a limitation the authors acknowledge.

Framing of the theoretical gap. I am less convinced that worst-case axiomatic failures are in "tension" with the success of post-training from human feedback methods, since the two can easily coexist. The paper is more persuasive as a study of when guarantees can be recovered (and not as an explanation of why RLHF works). Its own finding that changing the labels has little effect on the tested learned models supports this narrower reading.

Questions for authors
  • What practical guidance should readers takeaway from the majority results, given the discarded information and the limited benefit observed in the learned reward models?
Reviewer_odjJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies RLHF from the perspective of social choice theory. In this setting, the possible responses of an LLM to a prompt are the candidates, and the human preference rankings over responses correspond to voter preferences. Prior work has shown that the standard RLHF approach of learning a reward model based on the Bradley-Terry model is equivalent to aggregating voter preferences by the Borda count voting rule. There are theoretical drawbacks to Borda count, so this could be quite suboptimal. This paper studies this question and provides theoretical and empirical evidence to explain the success of RLHF in spite of these theoretical issues. On the theoretical side, the paper shows that replacing the Borda rule with the Copeland rule (based on scoring with the majority vote of pairwise preferences, rather than the winrate) would lead to much better social-choice axiomatic properties. The authors further observe that actual practical RLHF usually only uses a single rater sample for each pairwise comparison of responses, in which case it is claimed that the Borda and Copeland rules are equivalent and hence enjoy similar axiomatic properties.

The above results apply in the unrestricted setting, where the global optimal reward model is learned from the data. The paper also provides theoretical results in the setting where the reward model is restricted to being linear or a bounded degree polynomial. In these settings, if there is sufficient capacity (e.g. if the polynomial degree is above some threshold) then the model will achieve the global optimal reward and hence achieve the same axiomatic properties. If the capacity is more limited, the reward model learns a projection of the optimal reward onto some restricted set.

Finally, the empirical results in the paper show that the idealized unrestricted rewards under the two different rules match the Borda count and Copeland rules respectively. In this case, on prompts where the rules disagree, the Copeland rule has clearly better axiomatic properties. However, when using actual LLM reward models, there seems to be little difference between the two rules. One potential cause cited is capacity limitations of the learned rewards. A second cause identified in the paper is that on the RLHF datasets considered (e.g. ChatBot Arena) there are very few prompts where the Borda and Copeland rules actually differ, and when they do it is usually when the preferences between two options are very close to even across the raters. Thus, the differences between the rules have little impact on the outcome.

Critical strengths

The paper points out that in the practical RLHF setting with the Bradley-Terry model, the Borda rule is quite similar to the Copeland rule, which has much better social choice theoretic properties in general. This similarity could explain some of the success of standard RLHF whenever there is only one rater per preference ranking, or more generally on preference datasets where the two rules make similar predictions.

Critical weaknesses
  1. The writing in this paper leaves a lot to be desired, especially the mathematical writing. For example, the statement of Theorem 5.1 is very confusing, it is unclear what are the assumptions, and what are the conclusions. There are claims from another theorem (3.1) included as part of the theorem statement, which is not the correct way to write a theorem statement. The value k∗k^* is defined in the middle of the theorem statement. This is just one example, but such writing problems occur consistently in all the mathematical content throughout the paper. It is likely because an LLM was used to generate this content and the authors did not thoroughly check it. Basically all of the mathematical content requires significant rewriting.

  2. The paper I think misinterprets the translation between the sampled objective of RLHF and social choice rules. This is exemplified by the claim that if there is a single rater per pairwise comparison, then the Copeland rule and the Borda rule are equivalent. It is true that in this setting using the majority vote and the winrate for RLHF training are equivalent. However, the idea is that the RLHF dataset represents a random sample of human preferences over possible alternative responses, and that the expected value over such samples yields an objective function that is equivalent to the Borda count rule. The fact that in your sample you only have one rater per pairwise comparison does not mean that you are suddenly able to implement the Copeland rule. It actually means that you do not have sufficient sampled data to implement the true Copeland rule over the true distribution on human preferences. Thus this theoretical piece in the paper seems to hinge on an incorrect assumption.

  3. The results on limited capacity (linear or bounded degree) are standard calculations from classical convex optimization theory, and don't really seem to add much to the discussion.

  4. The experimental results are fairly limited. The synthetic experiments illustrate the linear setting well, but as mentioned above this is the least novel part of the paper. The LLM experiments have some issues e.g. the frozen LLMs with one trained linear layer really don't add much. There is only one experiment (the LoRA finetuned reward model) where an actual LLM is trained to produce rewards from human preferences. This is the whole point of the RLHF setup and of the connection to social choice theory, it deserves more than one single experimental result.

  5. The explanation of what is actually done in the experiments, and how we should interpret the results is also quite bad. Again it feels as if an AI has generated a lot of the text, and little editing has been done. This is not quite as bad as in the theoretical content, but still a significant impediment to understanding.

Overall I recommend rejection, basically all of the paper needs rewriting, especially the mathematical content. One of the most interesting theoretical claims about the equivalence of Borda and Copeland if there is a single rater per comparison is incorrect/does not make sense in the actual RLHF setting. The other theoretical results are quite standard and do not add much understanding, and the experimental results are limited.

Questions for authors

Please see above.

Reviewer_zrjjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the gap between social choice theory and how RLHF works in practice. The authors argue that the current formulation of using BTL style formulation satisifies pareto optimality but fails at the other axioms of social choice theory. They introduce an updated objective: instead of maximizing the probability of the majority binary label, maximize the win rate of the preferred response (and under this copeland rule it enforces all four axioms). The paper shows that if reward models are learned as linear or polynomial feature projections of LLMs, they have bounded error. On experiments on UF and ChatbotArena, the paper shows that the above label choice matter only in data limited regimes whereas model capacity is the bottleneck in practice.

Critical strengths
  • The paper presents important issues in existing approaches to alignment.
  • The paper studies changing the voting rule in RLHF to enable a more fair objective for aligning AI models.
  • The method also shows that fixing the candidate responses before choosing the polynomial degree overcomes past impossibility results.
  • The method is tested on UltraFeedback with 1.5B and 7B language models, and Chatbot Arena, showing that shared model capacity is the main bottleneck in practice.
Critical weaknesses
  • The paper is missing a key citation [1] -- it introduces a similar idea of using a copeland winner to train the models based on human preference data and shows why BTL fails as well. So, in the current presentation it is hard to understand the differences in contribution across the two papers. If the authors could clarify this it would be very helpful.
  • Because the main theory assumes a single fixed prompt while real reward models share weights across thousands of prompts, can the authors extend their guarantees or provide some insights about generalization across unseen prompts?
  • Could the authors clarify several experimental details needed for reproduction, such as how prompts are clustered in the capacity sweep and whether training the LoRA model for more than one epoch on the full dataset would help separate the two label types?

[1] A Mini-maximalist Approach to Reinforcement Learning from Human Feedback, Swamy et al.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.