- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work builds on Axioms for AI Alignment from Human Feedback (Ge et al., NeurIPS 2024). At a high level, it examines how the choice of preference aggregation method and the expressivity of the reward model jointly determine whether reward learning satisfies key social-choice axioms.
Critical strengths
An important and well-motivated problem. How alignment methods reconcile conflicting human judgments is an important question in its own right. I think the problem(s) the paper addresses are important.
An insightful connection to Copeland. The Copeland result is insightful and interesting in a way. The underlying voting rules are well known, but the point is still useful for anyone designing alignment methods.
Critical weaknesses
Relationship to established results. Ge et al. (2024) already studied majority-based loss functions. Once the unrestricted majority-label solution is identified with Copeland, the axiom guarantees follow directly from known properties of Copeland. The paper should make these connections more explict relative to prior work and their stated contributions.
The distinction between aggregation and expressivity has also been discussed before. For example, Section 3 of the survey by Halpern et al (2026) (AI Alignment From Social Choice Perspectives) identifies the aggregation rule that unrestricted BT implements, and then explains how restricting the reward class leads to further failures, including violations of unanimity. The paper should not present this distinction as a new conceptual contribution.
Scope of the guarantees. The theory assumes a fixed candidate set, balanced comparisons, and regularization on candidate scores. The experiments use models shared across prompts, but the main guarantees say nothing about how these models behave on unseen prompts or responses, a limitation the authors acknowledge.
Framing of the theoretical gap. I am less convinced that worst-case axiomatic failures are in "tension" with the success of post-training from human feedback methods, since the two can easily coexist. The paper is more persuasive as a study of when guarantees can be recovered (and not as an explanation of why RLHF works). Its own finding that changing the labels has little effect on the tested learned models supports this narrower reading.
Questions for authors
- What practical guidance should readers takeaway from the majority results, given the discarded information and the limited benefit observed in the learned reward models?