- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when minimizing a pairwise surrogate loss preserves unanimous preferences, termed Pareto optimality. It introduces Pareto calibration, a joint condition on the loss and comparison measure, and characterizes when every minimizing sequence eventually satisfies PO. The analysis distinguishes unanimous edges that participate in comparison cycles from those that can be satisfied without conflicting with other comparisons. It also develops sufficient conditions and proposes loss truncation and unanimous-edge reweighting as remedies.
Critical strengths
The main strength is that the paper's framework identifies a meaningful structural source of Pareto violations. Separating cyclic from acyclic unanimous edges makes the problem more interpretable: the difficult cases are those where enforcing a unanimous comparison competes with other comparisons.
Critical weaknesses
The main weakness is that some theoretical guarantees seem to fail under the stated assumptions. It would require at a minimum adding a differentiability assumption. In general, assumptions should be more clearly stated in the main body.
For hinge loss, take , , and , with all other weights zero. Despite satisfying monotonicity and , the unique minimizer modulo translation is , , violating PO. This contradicts Lemmas 4.2 and 5.2 as written.
Similarly, Theorem E.1 excludes zero minimizers, but equally weighted voters and , uniform spherical comparisons, and logistic loss give the unique optimum . This issue would remain even with a differentiability assumption.
The practical interventions are insufficiently specified and their benefits are evaluated too narrowly. The truncation guarantee depends on edge-specific caps, while the reweighting guarantee depends on sufficiently large unanimous-edge weights. However, the experimental description and the appendix do not specify these. More importantly, Figure 1 measures satisfaction of the same type of constraint the interventions explicitly prioritize. The paper does not report a held-out evaluation or performance on non-unanimous comparisons.
Questions for authors
Can you specify the truncation and reweighting settings and evaluate held-out prompts? Can you also report the fraction of prompt graphs that are Pareto-realizable and how infeasible graphs are handled?