- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper extends Ge et al (2024) by asking how much we need to depart from a linear reward representation in order to satisfy desirable social choice axioms. In particular, it focuses on PO and PMC, and adds (candidate-specific) corrections to the linear reward, minimizing their total size (in L1) subject to satisfying these axioms. I view this essentially as measuring the "cost of enforcing the axioms relative to a given representation" -- if the linear class cannot realize the required ranking, how much (candidate-specific) deviation from linearity is needed to make it possible?
Critical strengths
I think overall the problem is interesting and well motivated.
Critical weaknesses
-
While it is possible I did not follow the mathematical derivation correctly, I believe that some of the claims made in Section 4.1 are either incomplete or not correct when taken at face value. Specifically, the variable (line 293) is defined as a binary variable that equals one if . This is not the same thing as the number of reversed pairs -- a pair can be ranked in the correct order, with $0 < \langle \theta, x_a-x_b\rangle < \eta$ and still have . This seems to affect the interpretation of as KT distance throughout Section 4.1, including Lemmas 3--4.
-
My main conceptual pushback is about generalization, or more broadly the extent to which this problem is useful to study. In alignment, the reason we need to learn a reward function from a restricted hypothesis class is that the candidate space is effectively unbounded. We only observe preferences over a tiny set of responses, but the learned reward must ultimately score new responses. I view the linear model in Ge et al. (and the polynomial extension in Hollender and Kraiczy) mainly as tractable cases for studying this more general issue -- the reward model needs enough shared structure to generalize to unseen candidates, but that same restriction can constrain which aggregate rankings it can represent.
From this perspective, I am not sure how to interpret candidate-specific "slack" as addressing the same issue. Giving every observed candidate its own free correction makes essentially any finite ranking realizable, but those corrections contain no shared structure that tells us how to score a new candidate. In that sense, the "slack" solves the finite-sample representation problem partly through memorization, while the original alignment problem is precisely to find a representation that generalizes beyond the finite candidate set.
-
I found the motivation around Example 1 somewhat misleading -- the key issue in Ge et al is not that something like PO/PMC may be impossible to realize within the linear class -- a PO-respecting linear reward always exists. The negative result is that standard loss-based aggregation can nevertheless select a linear reward that violates PO. In other words, the issue is the aggregation rule, not a lack of representational capacity. Example 1 instead illustrates a different failure mode, where the desired ranking itself lies outside the linear class. I think the paper should distinguish these two issues more clearly in the introduction.
-
When the paper relaxes from a binary variable to a continuous variable in LP3, the resulting objective term is essentially hinge loss. This is already a well-known loss in this setting -- in particular, Ge et al explicitly consider hinge loss as a special case of their loss-based framework. So the hinge-loss component of LP3 does not itself seem conceptually new. The potentially new component is combining this loss with the candidate-specific slack and the preprocessing of the majority graph. I think the paper should clarify what this combination provides beyond standard hinge-loss optimization.
Questions for authors
-
Could you address my concern about the theoretical derivation in Section 4?
-
In Ge et al., PMC requires the output to agree with the majority ranking only when such a ranking exists and is realizable by a linear reward. In your model, the latter requirement seems to disappear because the candidate-specific slack makes any ranking over the observed candidates realizable. Could you clarify whether your definition of PMC is intentionally stronger than the one in Ge et al.? In particular, if the majority relation is acyclic but cannot be represented by any linear reward, Ge et al.'s PMC does not require the linear model to realize it, while your Def 1 does require the slack-augmented reward to realize it. If so, how should we interpret the statement that LP2 and LCPO satisfy the "same axioms"?
3 comments
Thank you for the response. Your clarification on PMC makes sense to me and resolves my concern. On Section 4, I agree that the issue looks fixable. I would just clarify when a correctly ranked pair can still count as a violation.
More broadly, I see the value of asking how much the scores need to change to satisfy the axioms on a given set of candidates. What is still unclear to me is whether learning those corrections helps the model rank new responses.
Thank you for the thoughtful response.
Finding generalization using the slack varaibles/correction is the next direction of research. We have already shown in Section 4.2 how the correction variables help us find a certificate of realizability based on the feature geometry. It should also be noted that in Proposition 4, we have shown that using the dual variable of , i.e., helps us to get the linear realizability characteristic based on the feature values (feature-balanced flow). So, finding the characteristic of the feature geometry will give a much stronger result than Hollender et al. (2026).
We thank the reviewer for useful feedback, comments, and careful reading.
Theoretical derivation in Section 4: Correct. In the ILP, Eqn 7 should, ideally, be uabL>−⟨θxa−xb⟩. This does not affect the proof of Lemma 3 and Lemma 4. We will update the equation.In LP3, for implemneting the strict inequality we write, uabL≥η−⟨θxa−xb⟩. Note that η is a hyperparameter and we can set η<κ as defined in Section 4.2. Then, there cannot be a pair (a,b) such that 0<⟨θ,xa−xb⟩<η.
Generalization: The augmented reward (θ,ε) satisfies PO and PMC exactly on the observed comparisons, with no assumption on the voters or the feature geometry. No loss-based rule, including BTL, offers this guarantee. The linear part θ is the deployable object, and the held-out results characterize how it transfers to new prompts. Extending exact axiom satisfaction beyond the observed comparisons is the natural next question. Our framework makes it precise, since the slack identifies the generalizing representation of which candidates would need to be corrected. We conjecture that on unrestricted data, the performance of our model could be comparable to that of loss-based models, e.g., BTL. However, as shown in Hollender et al., the main open question is identifying restrictions on preferences when the performance of loss-based models and our model differ.
Candidate-specific slack: Since we have no assumptions about the behavior of the data, it may be that no linear reward model can produce the ranking required to satisfy the PO and majority constraints. Previous work by Ge et al. and by Hollender and Kraiczy both assumes restrictions on the data. Even if we assume that there is a ground truth θ0 and the required ranking can be modeled by it, the data might have noise, and thus we don't get a linearly realizable input. The slack values let us obtain a θ close to θ0.
Example 1: We do not claim in Example 1 that Ge et al. fail in the linear class. However, the human preferences may not be in the linear class. Our main contribution is to design a rule that can be applied even when the preferences cannot be modeled by a linear reward function. To understand the difference between our rule and Ge et al., consider this: The first step of leximax Copeland in Ge et al. is "It ranks first the candidate with the highest Copeland score that can be feasibly ranked first under some parameter vector θ." Now consider the execution of this step on Example 1 where the human feedback is b>c>a. There is no value of θ∈R such that b comes first, as explained in Example 1. Thus, PO is not satisfied. We have explained it at the end of Section 3. This also shows that a PO-respecting linear reward does not always exist. We will clarify it in the introduction.
LP3 and Hinge loss are not the same. LP3 uses a scaled pairwise hinge-loss penalty, with an additional optimization objective minimizing slack and explicitly enforcing PO and PMC using corrected rewards. The standard pairwise hinge loss for a PO or majority pair ℓhinge(θ;a,b)=max0,η−⟨θ,xa−xb⟩. This loss penalizes a pair whenever the candidate a does not achieve the required margin η over candidate b. In LP3, the relaxed variable uab≥0 satisfies Luab≥η−⟨θ,xa−xb⟩. Consequently, LP3 can be rewritten as: minθ,ϵ∑i=1m∣ϵi∣+Lλ∑(a,b)∈Pℓhinge(θ;a,b) s.t ⟨θ,xa−xb⟩+(ϵa−ϵb)≥η,∀(a,b)∈P, and the remaining constraints of LP3. In other words, LP3 can be viewed as a regularized pairwise hinge loss formulation, where the regularization is not simply an L1 or L2 penalty on θ. Instead, it minimizes the L1-norm of candidate-wise slack ϵ, while requiring those corrections to ensure the axioms. We will add the relation to hinge loss for clarification. Furthermore, our experiments showed that linear hinge loss performs worse than LP3. We added this in the experiments as well (See reply for reviewer CW6f).
PMC Definition: You are correct that ours is stronger. An axiom definition should not be model dependent. The PMC axiom in Ge et al. assumes the output ranking is linearly feasible, i.e., the human preference profile is linearly realizable. We intentionally write the standard definition, which is independent of the model. The claim that LP2 and LCPO satisfy "the same axioms" is removed. The accurate statement is: LCPO's linear reward satisfies conditional PMC; LP2's augmented reward satisfies unconditional PMC; and LP2's slack is zero exactly when its linear part alone suffices at margin η.