- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PECCO for multi-objective alignment when preferences over objectives may be specified by an ordering, numerical weights, or both. It represents acceptable weights as a convex set, uses that set to constrain gradient combinations, and proves convergence to a preference Pareto stationary point under smoothness assumptions. Experiments on summarization and helpfulness–harmlessness tasks examine objective trade-offs across several model families.
Critical strengths
- The admissible preference set is a clear way to combine partial objective orderings with optional numerical preferences.
- PECCO connects this formulation to a tractable multi-gradient update and provides a convergence result.
Critical weaknesses
-
Table 3 does not measure gradient conflicts on the same basis for all methods: RACO and DPO-LW are checked against the original objective gradients, whereas PECCO is checked against gradients transformed by its admissible preference set. Thus the near-zero conflict rates do not, by themselves, establish a comparative reduction in conflicts with the original objectives.
-
The evidence for broader use remains limited. For general partial orders, the extreme-point representation may also become costly as the number of objectives grows.
Questions for authors
-
Can the authors report conflict rates for all methods using both the original objective gradients and the transformed gradients? This would clarify how much of PECCO’s reported advantage in Table 3 comes from the choice of reference gradients.
-
How do the number of extreme points and the optimization cost scale with larger, partially ordered objective sets? A runtime example beyond three objectives would help assess the claimed flexibility.
Q1: Can the authors report conflict rates for all methods using both the original objective gradients and the transformed gradients? This would clarify how much of PECCO’s reported advantage in Table 3 comes from the choice of reference gradients.
However, conflict rates on F are not suitable for conflict-avoidance comparison in PECCO. First, PECCO handles a different optimization problem with extra ordinal information, confidence levels, and inner structure. This problem turns out to involve the transformed objects in H, for which we compute their gradient conflict rates. The old metric, accounting only for cardinal information, can therefore be biased.
Moreover, gradient conflicts are dynamic, primarily making substantial impacts on ongoing training (e.g., training instability, mutual gradient interference, and cancellation). Thus, a reasonable metric should capture such dynamic effects of conflicts, potentially motivating qualitative explanations of the consequences that conflicts can actually cause. This leads to different metric choices for RACO/DPO-LW and PECCO, and undermines the rationale for measuring gradient conflict uniformly based on F.
Q2: How do the number of extreme points and the optimization cost scale with larger, partially ordered objective sets? A runtime example beyond three objectives would help assess the claimed flexibility.
Generally, the number of extreme points increases with the complexity of the objective set, but the exact situation needs case-by-case discussion. People may have more objectives while applying more constraints to the admissible sets. The challenges of discussing this include, but are not limited to, the following: redundant constraints should add no extra extreme points; polytope tricks applied to the admissible set may reduce the number of necessary extreme points through approximation. Furthermore, the granularity of the admissible set should be decided in practice, according to actual needs and environmental constraints. Although a comprehensive evaluation of many-objective scalability is too lengthy to include here, Appendix B.2 offers some preliminary discussion of this, and more hands-on tricks for customizing efficient polytope V-representations can be found in polytope theory articles and their references.