acceptodds
Under review as a conference paper at ICLR 2027

Preference-aware Multi-objective Alignment

Abstract

Alignment is an important component of post-training for modern foundation models and often involves multiple conflicting objectives with heterogeneous preferences specified by stakeholders. Failing to account for these conflicts and preferences can lead to undesirable trade-offs and unstable optimization. In practice, however, objective-level preferences may be only partially specified, e.g., through qualitative orderings among objectives rather than precise numerical weights. To address this challenge, we propose a preference-aware multi-objective alignment framework that models preferences over objectives through an admissible preference set, enabling a unified representation of both cardinal and ordinal preference information, including partial order constraints. Building on this framework, we develop PrEference-Consistent Cardinal-Ordinal Alignment (PECCO), an algorithm that mitigates gradient conflicts among objectives while preserving the specified preferences throughout optimization. We further establish convergence to a preference-aligned Pareto stationary point. Experiments on standard benchmarks show that PECCO achieves performance comparable to existing multi-objective preference-alignment methods while providing greater flexibility in accommodating diverse forms of objective-level preferences.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language model alignment, Direct preference optimization, Multi-objective optimization, Preference uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3yvwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PECCO for multi-objective alignment when preferences over objectives may be specified by an ordering, numerical weights, or both. It represents acceptable weights as a convex set, uses that set to constrain gradient combinations, and proves convergence to a preference Pareto stationary point under smoothness assumptions. Experiments on summarization and helpfulness–harmlessness tasks examine objective trade-offs across several model families.

Critical strengths
  1. The admissible preference set is a clear way to combine partial objective orderings with optional numerical preferences.
  2. PECCO connects this formulation to a tractable multi-gradient update and provides a convergence result.
Critical weaknesses
  1. Table 3 does not measure gradient conflicts on the same basis for all methods: RACO and DPO-LW are checked against the original objective gradients, whereas PECCO is checked against gradients transformed by its admissible preference set. Thus the near-zero conflict rates do not, by themselves, establish a comparative reduction in conflicts with the original objectives.

  2. The evidence for broader use remains limited. For general partial orders, the extreme-point representation may also become costly as the number of objectives grows.

Questions for authors
  1. Can the authors report conflict rates for all methods using both the original objective gradients and the transformed gradients? This would clarify how much of PECCO’s reported advantage in Table 3 comes from the choice of reference gradients.

  2. How do the number of extreme points and the optimization cost scale with larger, partially ordered objective sets? A runtime example beyond three objectives would help assess the claimed flexibility.

1 comment

Q1: Can the authors report conflict rates for all methods using both the original objective gradients and the transformed gradients? This would clarify how much of PECCO’s reported advantage in Table 3 comes from the choice of reference gradients.

WeightsRACO Original FFDPO-LW Original FFPECCO original FF, Conf =0.4=0.4PECCO original FF, Conf =0.8=0.8PECCO transformed HH, Conf =0.4=0.4PECCO transformed HH, Conf =0.8=0.8
(0.20,0.80)(0.20, 0.80)0.77%10.99%0.92%7.04%0.00%0.00%
(0.35,0.65)(0.35, 0.65)0.07%1.83%0.99%1.27%0.00%0.00%
(0.50,0.50)(0.50, 0.50)0.07%0.85%0.70%0.28%0.14%0.00%
(0.65,0.35)(0.65, 0.35)0.28%3.87%2.18%3.87%0.00%0.00%
(0.80,0.20)(0.80, 0.20)1.55%9.65%3.59%7.54%0.00%0.00%

However, conflict rates on FF are not suitable for conflict-avoidance comparison in PECCO. First, PECCO handles a different optimization problem with extra ordinal information, confidence levels, and inner structure. This problem turns out to involve the transformed objects in HH, for which we compute their gradient conflict rates. The old metric, accounting only for cardinal information, can therefore be biased.

Moreover, gradient conflicts are dynamic, primarily making substantial impacts on ongoing training (e.g., training instability, mutual gradient interference, and cancellation). Thus, a reasonable metric should capture such dynamic effects of conflicts, potentially motivating qualitative explanations of the consequences that conflicts can actually cause. This leads to different metric choices for RACO/DPO-LW and PECCO, and undermines the rationale for measuring gradient conflict uniformly based on FF.

Q2: How do the number of extreme points and the optimization cost scale with larger, partially ordered objective sets? A runtime example beyond three objectives would help assess the claimed flexibility.

Generally, the number of extreme points increases with the complexity of the objective set, but the exact situation needs case-by-case discussion. People may have more objectives while applying more constraints to the admissible sets. The challenges of discussing this include, but are not limited to, the following: redundant constraints should add no extra extreme points; polytope tricks applied to the admissible set may reduce the number of necessary extreme points through approximation. Furthermore, the granularity of the admissible set should be decided in practice, according to actual needs and environmental constraints. Although a comprehensive evaluation of many-objective scalability is too lengthy to include here, Appendix B.2 offers some preliminary discussion of this, and more hands-on tricks for customizing efficient polytope V-representations can be found in polytope theory articles and their references.

Reviewer_CZZSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PrEference-Consistent Cardinal-Ordinal Alignment (PECCO), a multi-objective alignment framework that incorporates both numerical (cardinal) and ranking-based (ordinal) preferences over alignment objectives.

It It introduces an admissible preference set that combines ordinal constraints with a prescribed numerical preference vector through a confidence parameter. The authors characterize this set using an extreme-point representation, allowing them to reformulate the constrained multi-objective optimization problem over a probability simplex. PECCO extends the Multiple Gradient Descent Algorithm (MGDA) by restricting gradient-combination weights to the admissible preference set. It is shown that the algorithm reaches an approximate preference Pareto stationary point with O(\epsilon^{-2}) iteration complexity.

Experiments on multi-objective LLM alignment tasks, including quality, conciseness, and faithfulness, demonstrate that PECCO achieves competitive performance against existing frameworks.

Critical strengths

1.the admissible preference set provides an elegant way to combine ordinal constraints, numerical preferences, and uncertainty. Its convex geometric interpretation is intuitive.

  1. Representing the admissible set through its extreme points allows the optimization to be expressed as a simplex-constrained problem. This facilitates the use of existing multi-gradient optimization techniques.

  2. The paper provides a convergence guarantee for the proposed algorithm under smoothness and lower-boundedness assumptions, which strengthens its theoretical foundation.

Critical weaknesses
  1. The main algorithmic contribution appears to be an application of MGDA to a restricted convex set of gradient-combination weights. Specifically, PECCO replaces the standard simplex constraint in MGDA with the proposed admissible preference set. Eventually, the resulting problem becomes a standard MGDA-type optimization problem over transformed objectives. While the preference representation is useful, the optimization procedure itself appears to follow directly from existing techniques.

  2. The experiments primarily compare PECCO against RACO and DPO-LW. However, these baselines do not directly isolate the benefit of the proposed admissible preference set. Comparison should also be done with standard MGDA with explicitly imposed ordinal weight constraints, without the proposed extreme-point transformation, as well as simpler approaches that optimize weighted objective combinations under ordinal constraints. Such comparison are needed to determine whether the improvements arise from the proposed algorithm or simply from constraining the feasible gradient-combination weights.

  3. The number of extreme points of a general preference polytope may grow rapidly with the number of objectives and the complexity of the preference constraints. Complexity analysis of constructing the admissible preference set and solving the associated optimization problem would be benificial. Experiments involving larger numbers of objectives would help establish the scalability of the proposed framework.

Questions for authors
  1. How does PECCO differ fundamentally from applying standard MGDA to the transformed objectives defined by the extreme points of the admissible preference set? 2.How does the computational complexity scale with the number of objectives and the number of extreme points in the admissible preference set?
  2. How sensitive is the method to inconsistent numerical and ordinal preference specifications?
1 comment

Response to weakness:

For the first weakness you indicate, we need to clarify that PECCO mainly focuses on preference consistency and conflict avoidance in user-preference alignment, rather than proposing theoretically new optimization algorithms from scratch. Directly imposing the same preference constraints in MGDA yields an equivalent optimization subproblem, since our extreme-point transformation is an exact reparameterization. Moreover, ordinal-only constrained MGDA corresponds to the λ=0\lambda=0 in Table 4. To our best known, there is no simpler multi-objective alignment methods that target weighted objective combinations under ordinal constraints, and both DPO-LW and RACO can be considered as the approximation to ordinal constraints by numerical vectors. Moreover, our confidence ablation varies the admissible preference region while retaining the same optimization procedure, examining its effect on the resulting trade-offs. Our contribution is precisely the integration of admissible preference modeling with conflict-aware optimization, rather than an improvement attributable to the extreme-point transformation itself. The number of extreme points can indeed increase with more objectives and constraints, but an appropriate granularity strategy can help restrict this number by approximating the original admissible set. In our view, this granularity should therefore be adaptable—that is, it can be adjusted according to practical needs and computational constraints.

Q1: How does PECCO differ fundamentally from applying standard MGDA to the transformed objectives defined by the extreme points of the admissible preference set? How does the computational complexity scale with the number of objectives and the number of extreme points in the admissible preference set?

They are mathematically equivalent as shown in Appendix B.2. Given the KK objective gradients, each projected-gradient iteration can be implemented in O(dK+KM+Mlog⁡M)O(dK+KM+M\log M) time using matrix–vector products and a sorting-based simplex projection.

Q2: How sensitive is the method to inconsistent numerical and ordinal preference specifications?

PECCO addresses uncertainty in numerical preferences, rather than contradictory specifications. We require the numerical preference, when provided, to satisfy the ordinal constraints, i.e., w~∈Pord\tilde w\in\mathcal P_{\mathrm{ord}}, ensuring ordinal consistency for all confidence levels. We will make this compatibility requirement explicit.

Reviewer_6W9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to incorporate incomplete preference information, including numerical weights and objective priorities, into multi-objective model alignment. It represents ordinal preferences through linear ordering constraints on the probability simplex and combines them with a numerical preference vector through a confidence parameter to construct an admissible preference set. Under smoothness and lower-boundedness assumptions on deterministic objectives, it establishes a complexity guarantee for reaching an approximate preference Pareto stationary point. Experiments on summarization and safety alignment evaluate preference trade-offs, weight consistency, and gradient conflicts.

Critical strengths
  1. The preference representation is more flexible than a single weight vector. The admissible preference set accommodates complete rankings, partial rankings, and numerical preferences with varying confidence levels, making it suitable for scenarios where precise objective weights are difficult to specify.
  2. The framework clearly connects modeling, optimization and analysis. Through an extreme-point representation, the method reformulates preference constraints as an MGDA-type problem over transformed objectives and establishes a preference-stationarity guarantee. Experiments span multiple models, two- and three-objective settings and both summarization and safety alignment, demonstrating applicability across different alignment scenarios.
Critical weaknesses

The conflict-avoidance guarantee and the empirical comparison use different reference gradients. The minimum-norm argument in Section 5 guarantees a descent direction for all admissible gradient combinations, but generally does not guarantee descent for every original objective. In Table 3, conflict rates for RACO and DPO-LW are measured against the original objective gradients, whereas those for PECCO are measured against the transformed-objective gradients.

Questions for authors
  1. Table 3 uses different reference gradients for different methods. Can you report conflict rates for all methods against both the original and transformed objective gradients using a consistent evaluation protocol? In particular, what is PECCO’s conflict rate with respect to the original objective gradients?
  2. The common descent guarantee in Section 5 appears to apply only to admissible gradient combinations, rather than guaranteeing simultaneous descent for all original objectives. Can you clarify the scope of this guarantee and explain how it relates to preference consistency in generated outputs?
3 comments

Q1: Table 3 uses different reference gradients for different methods. Can you report conflict rates for all methods against both the original and transformed objective gradients using a consistent evaluation protocol? In particular, what is PECCO’s conflict rate with respect to the original objective gradients?

We report PECCO’s conflict rates with respect to the original objective gradients below.

WeightsRACO Original FFDPO-LW Original FFPECCO (original FF), Conf =0.4=0.4PECCO (original FF), Conf =0.8=0.8PECCO (transformed HH), Conf =0.4=0.4PECCO (transformed HH), Conf =0.8=0.8
(0.20, 0.80)(0.20,\,0.80)0.77%10.99%0.92%7.04%0.00%0.00%
(0.35, 0.65)(0.35,\,0.65)0.07%1.83%0.99%1.27%0.00%0.00%
(0.50, 0.50)(0.50,\,0.50)0.07%0.85%0.70%0.28%0.14%0.00%
(0.65, 0.35)(0.65,\,0.35)0.28%3.87%2.18%3.87%0.00%0.00%
(0.80, 0.20)(0.80,\,0.20)1.55%9.65%3.59%7.54%0.00%0.00%

However, conflict rates on FF are not suitable for conflict-avoidance comparison in PECCO. First, PECCO handles a different optimization problem with extra ordinal information, confidence levels, and inner structure. This problem turns out to involve the transformed objects in HH, for which we compute their gradient conflict rates. The old metric, accounting only for cardinal information, can therefore be biased.

Moreover, gradient conflicts are dynamic, primarily making substantial impacts on ongoing training (e.g., training instability, mutual gradient interference, and cancellation). Thus, a reasonable metric should capture such dynamic effects of conflicts, potentially motivating qualitative explanations of the consequences that conflicts can actually cause. This leads to different metric choices for RACO/DPO-LW and PECCO, and undermines the rationale for measuring gradient conflict uniformly based on FF.

Q2: The common descent guarantee in Section 5 appears to apply only on admissible gradient combinations, rather than guaranteeing simultaneous descent for all original objectives. Can you clarify the scope of this guarantee and explain how it relates to preference consistency in generated outputs?

It is intentional that the guarantee applies to all admissible gradient combinations, rather than every original objective individually. Conflicts among objectives generally preclude simultaneous descent of all objectives throughout the optimization trajectory, making each update inherently a trade-off.

Moreover, preference consistency in generated outputs is not determined solely by the optimization trajectory, but can also depend on datasets, tasks, relative objective scales, and other factors. Thus, an optimization-side method can only directly control part of this process. PECCO focuses on this controllable part by making each trade-off decision preference-aware throughout the optimization trajectory.

Accordingly, the guarantee in Section 5 should be interpreted as an optimization-level guarantee over the admissible preference set, rather than a guarantee of simultaneous descent for all original objectives or exact preference consistency of final outputs.

Thank you for your clarifications and additional results. I appreciate your efforts to address my concerns. However, the theoretical guarantee does not directly establish preference consistency in generated outputs. While I recognize the value of the proposed framework, my main concerns remain insufficiently resolved. Therefore, I will keep my original score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.