acceptodds
Under review as a conference paper at ICLR 2027

Multi-Objective LLM Alignment using Multivariate Stochastic Dominance

Abstract

Though Large Language Models (LLMs) are increasingly performant on a wide range of tasks, multi-objective alignment with a variety of evaluative signals is a key challenge. Existing methods typically optimize scalar preference signals, seek to balance competing objectives, or learn policies spanning different trade-offs, without explicitly targeting joint improvement over a reference policy. In this paper, we introduce Stochastically Dominant Multi-Objective Alignment (SDMA), a framework for multi-dimensional LLM alignment without committing to particular trade-offs between evaluative dimensions. Instead, we employ first-order multivariate stochastic dominance to guide improved performance for all monotonic functions of these different dimensions. We demonstrate balanced improvement in our results for multiple dimensions, including subsets of helpfulness, harmlessness, humor, honesty, and instruction-following.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reinforcement learning from human feedback (RLHF), Distributional alignment, Multivariate stochastic dominance, LLM alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QvSZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to improve several LLM reward functions simultaneously relative to a fixed reference policy. Its target is multivariate stochastic dominance: improvement for every coordinatewise monotone utility of the joint reward vector. The proposed method, SDMA, uses optimal transport to match samples from the current and reference policies, penalizes reward shortfalls, and converts the matching into a weighted-best-match policy update. Experiments use Llama-2-7B model and HH-RLHF dataset with two to five objectives. Compared with multiple alignment baselines, SDMA reports higher empirical dominance measures and positive improvements across all five reward functions in the main experiment.

Critical strengths
  1. A meaningful alignment goal. Definition 2 asks for improvement over a reference policy under every coordinatewise monotone utility. This is a useful goal when several response qualities matter and the user's utility function is not known in advance. It gives the paper a clear motivation for studying the joint reward distribution.

  2. The reported gains are substantial. In the five-objective setting, Table 1 gives an MPD of 94.1% for SDMA and 22.1% for the best trained baseline on this metric. SDMA also has positive effect sizes on all five objectives. Table 2 reports strong dominance metrics across two to five objectives. The size and consistency of these gains make the method worth further study.

Critical weaknesses
  1. The policy update needs a derivation. Theorems 3 and Theorem 4 give structural results from prior work. The missing step is the connection of Eq. (5) to Algorithm 1, line 4. The paper does not explain why this update can reduce the transport cost. I would like to see the surrogate being optimized and a derivation of the coefficients aia_i. There is also a distinction between matching finite samples and establishing dominance between the population reward distributions.

  2. The experiments leave the practical advantage uncertain. Work on conflicting objective gradients includes MGDA [1], CAGrad [2] and subsequent approaches for LLM alignment, such as RACO [3] and GAPO [4]. These are highly related to the idea of simultaneous improvement for multiple objectives. Comparing with these methods would help establish how much SDMA gains over approaches that explicitly address gradient conflicts.

  3. The paper gives no training time, GPU-hours, or training OT solver, so the overhead of the transport step is difficult to assess. Reward normalization and the numerical settings of α\alpha and β\beta also need to be specified. Finally, the five generation trials in Section 4.3 measure variation in sampled responses; they do not show whether the results are consistent across independent training runs.


References

[1] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique, 350(5–6):313–318, 2012.

[2] Bo Liu et al. Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS, 2021.

[3] Peter Chen et al. Reward-free Alignment for Conflicting Objectives. ICML, 2026.

[4] Chengao Li et al. Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models. ACL, 2025.

Questions for authors
  1. Could you derive the weighted-best-match update in Algorithm 1 and identify the objective optimized by the implementation? Please explain how it relates to the transport objective when responses are resampled and the matching changes.

  2. Could you compare SDMA with existing approaches that explicitly address gradient conflicts under a fair setup, e.g., with comparable compute budgets?

  3. Please report wall-clock training time and the OT solver, including the fraction of runtime spent constructing and solving the OT problems. How does the comparison change under similar total compute? Results from multiple independent training seeds would also help assess the reliability of the reported gains.

  4. Please discuss Kaya et al. [5], which also studies policy improvement using stochastic dominance in multi-objective RL. How do the dominance criterion, choice of reference policy, and LLM setting distinguish the contribution here?


References

[5] Ege Can Kaya et al. Stochastic Dominance Driven First-Order Policy Optimization for Multi-Objective Reinforcement Learning. UAI, 2026.

3 comments
AuthorsOpenReview ↗

Clarifications, positioning, and novelty

Thank you for your clarification questions! We hope our responses help distinguish our approach from existing work.

Q1. Explanation of policy gradient updates.

For each demonstration we sample multiple outputs, and each demonstration is matched to multiple outputs. Our policy gradient updates put more gradient weight on the sampled outputs that are more desirable matches, i.e., (closer to) Pareto-dominant with the paired demonstration. Line 3 of our algorithm downweights the worst matched samples. We find empirically that this accelerates the our algorithm towards a better policy during training. We suspect that like policy gradient update weighting in general (Sutton and Barto, 2018), there are many weights that provide convergence, but vary in how quickly.

Q2. Comparison with gradient conflict methods.

The methods in [1]-[4] seek single points/tradeoffs in the competing objectives. Our approach (and distributional approaches more generally) seek a distribution over outputs that reflect many different tradeoffs across multi-objectives. This is generally required to achieve stochastic dominance because each demonstration may reflect different trade-offs, so a single point/tradeoff will not be flexible enough to achieve stochastic dominance (and provide corresponding monotonic utility guarantees).

Q3. Run Time

Computational cost. The original SDMA runs took 3.2 to 4.3 hours on one NVIDIA H200 NVL GPU. Most of that time was spent generating responses and scoring them with the reward models. At each training step, we use MOSEK to solve a quadratically regularized optimal transport problem matching 30 reference responses to 90 policy responses pooled across prompts. Constructing and solving this problem took approximately 0.06 seconds per step. Including both training and validation matching, optimal transport accounted for less than 1% of wall-clock time. Appendix G reports training times for all methods (Table 16) and the breakdown by phase for SDMA (Table 17).

Comparison at matched training time. For each objective count, we used DPO's training time as the budget and selected the best validation checkpoint reached within that time for each of two independent SDMA training seeds. We evaluated both methods on the test prompts using the original reward models. The table compares mean matched Pareto dominance (MPD, in percent) across the two SDMA seeds with the final DPO checkpoint.

Number of objectives          2       3       4       5
Training budget (hours)    0.52    0.77    1.03    1.29
SDMA MPD                   91.4     6.5    80.5    28.8
DPO MPD                    88.3     2.3    25.0    16.7

SDMA achieved higher MPD at each matched training budget, but the shorter budget substantially reducedsSDMA's MPD relative to full runs, particularly with 3 and 5 objectives. With 3 objectives, both methods had low MPD. As detailed in Appendix G, these timings exclude one-time data preparation for both methods.

Reward normalization and hyperparameters. We use raw rewards with 2 objectives and standardized rewards using SFT reference statistics with 3 to 5 objectives. The objective weights are fixed at uniform values with 2 objectives and learned from a uniform initialization with 3 to 5. Appendix F (Table 14 and accompanying text) gives the numerical weights and margins, and explains how the margins were chosen.

Training stability. We train SDMA with three independent seeds at each objective count. For each seed, we average five generation trials before computing the mean and standard deviation across training seeds. Under the original reward models, MPD varies little across seeds for 2 to 4 objectives. The five-objective setting showed more variation, with a mean of 89.9% and a standard deviation of 4.3%. Appendix E (Tables 10–13) reports the results for each objective count. Each baseline was trained with 1 seed.

Q4. Discussion of Kaya et al.

Kaya et al. (2026) considers the distributional definition of stochastic dominance based on the cumulative density function (in their paper). Eq. (2) in our paper is the equivalent right-sided version of this (i.e., defined by upper sets). The utility-based definition that guides our optimization (Def. 2) is stronger, as described in lines 194-197, so the Kaya et al. approach does not provide (approximate) performance guarantees for the same broad class of monotonic utility functions that are the focus of our guarantees.

We do not commit to any particular trade-offs in the multiple objectives under our approach and instead require improvement for all monotonic utility functions. This generally requires a stochastic policy that has good coverage of different demonstrations. The LLM setting is challenging (compared to previous work in sequential decision making with two or three main modes of demonstrated behavior) because the reference policy may already produce a distribution over a variety of outputs.

Thank you for the clarifications and additional results. My concern in Q3 and Q4 have been resolved.

However, for Q1, the response does not provide the requested derivation connecting the update to the transport objective.

For Q2, I understand that methods [1]-[4] do not target stochastic dominance. However, I suppose your ultimate goal is to improve performance in the multiple objectives context. Stochastic dominance is just one way to pursue this goal, and other ways may be better or worse. A comparison would therefore help understand whether SDMA performs better than these alternatives.

Thus, I would like to keep my score 2.

Thank you for your quick feedback.

[4] is similarly applied to LLMs and its goal (avoid unbalanced optimization of multiple objectives) is most similar to ours. However, it has only been evaluated using two reward dimensions (helpfulness vs. harmlessness). To obtain a diverse frontier (Figure 3), which is required to compete on our evaluation measures, it uses various weightings between the two reward dimensions {0.2, 0.4, 0.6, 0.8} to train separate models. Our largest experiments have five reward dimensions. The approach in [4] does not scale in a reasonable way to provide comparisons that would be competitive against our existing baselines.

Reviewer_asYQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SDMA fine-tunes an LLM against K reward models (K = 2-5: helpfulness, harmlessness, humor, honesty, instruction-following) without fixing weights between them. It trains the policy so that its reward vector first-order stochastically dominates that of the SFT reference, which means its expected utility is at least the reference's for every monotone utility of the K rewards (Def. 2, Eq. 1). The loss is a subdominance cost matched by optimal transport (OT) (Eqs. 3-4), minimized with a REINFORCE update (Algorithm 1). On Llama-2-7B / HH-RLHF, SDMA scores highest on all dominance metrics for each K against seven baselines (e.g., Matched Pareto Dominance (MPD) 94.1 vs 22.1 at K = 5, Table 1).

Critical strengths
  1. Consistent gains on every dominance metric. Among non-oracle methods, SDMA leads on all dominance metrics for K = 2-5, by margins far above the base-vs-base noise control (Tables 1-5). At K ≥ 3 it also beats best-of-16 selection from SFT, which picks with the same reward models, by 32-88 MPD points (Tables 1, 4, 5).

  2. No regression on any objective. SDMA is the only method whose per-objective reward quantiles stay at or above SFT on every objective (Figs. 2-5). Each baseline falls below SFT somewhere, for example DPO on helpfulness and RiC on harmlessness at K = 5.

The no-regression result (Strength 2) is useful for multi-objective alignment. Apart from Theorem 4 (Weakness 2), I found no errors in how the method transfers PSD. The open question is reward hacking (Weakness 1), which further experiments can potentially settle.

Critical weaknesses
  1. The results do not rule out reward hacking. The paper lists one set of reward models for its objectives (l.282-287) and calls them the evaluation reward models (l.313), which indicates that the models SDMA optimizes also produce all reported metrics. The risk here comes from SDMA's update, which has no KL term (Algorithm 1, line 4), and the paper reports KL for no method. SDMA can therefore move further from SFT and exploit the reward models, several of which are small (GPT-2-large for helpfulness and harmlessness, a RoBERTa humor classifier). The authors name this risk and propose redundant reward models from different sources as a remedy (l.482-485), but do not run it.

  2. Low novelty, and the transferred theory is not extended. SDMA applies PSD (Farajzadeh et al., NeurIPS 2025), an imitation-learning method, to LLM fine-tuning: Eq. 3 is PSD Eq. 7, Eq. 4 is PSD Eq. 8, and Algorithm 1 is PSD's update with the weights of PSD Eq. 11. The paper credits PSD for each (l.141-145, 212, 234, 245), and the transfer of PSD replaces trajectory features with reward-model scores and demonstrations with SFT samples. Theorem 4 is credited to PSD (it corresponds to PSD Thm 3.6) but concludes a stronger order: PSD proves dominance only over conical sums of rewards, and the paper does not prove the utility-based version of Def. 2 itself. SDPO (AAMAS 2026), which already learns a policy that stochastically dominates a reference policy from user rankings (per ranker, on a toy domain), is not cited.

Most hyperparameters (α, β, learning rate, number of updates, baseline settings) are missing, and the paper does not say whether OT matching is within or across prompts (l.276-277). The results do not rule out reward hacking, and the method is PSD applied to a new setting without extending its theory. If either experiment above shows that SDMA's advantage holds, I would consider raising my rating to weak accept.

Questions for authors
  1. (Experiment; could change my rating.) Can you evaluate all methods with held-out reward models not used in training (e.g., a different reward-model family for helpfulness and harmlessness)?

  2. (Experiment; could change my rating.) Can you add a KL term to SDMA, or report KL to SFT for every method and compare at matched KL?

1 comment
AuthorsOpenReview ↗

Reward hacking and KL divergence evaluations

Thank you for your insightful questions! We have added experiments that address your concerns (reward hacking/overfitting).

Q1 Reward Hacking?

Our new evaluations with withheld rewards show our performance generalizes to other withheld rewards. We evaluate SDMA and all seven baselines using DPA for helpfulness and negated Beaver cost for harmlessness (neither used during training). In the two-objective setting, SDMA improves with respect to SFT on both held-out measures (+0.45 helpfulness; +0.77 harmlessness; SDMA and DPO tied at 99.8% matched Pareto dominance: Appendix D, Table 7).

SDMA yields 96.4%, 97.9%, and 97.7% matched Pareto dominance for K=3,4,5K=3,4,5 compared to 80.0%, 96.8%, and 94.8% for DPO (Appendix D, Tables 3, 8-10), which was the strongest baseline at each KK. These results show helpfulness and harmlessness gains transfer to the held-out judges we report.

Q2 KL Divergence

We measured each policy's KL divergence to SFT in nats per response. Each column below gives the number of training objectives.

Method2345
SDMA76.961.567.137.3
DPO22.230.329.844.1
AOT43.845.482.049.3
Rewarded Soups3.21.20.80.5
MODPO12.35.66.49.5
RiC26.125.532.020.0
MaxMin-RLHF1.41.61.31.2
MORLHF1.03.32.11.2

At two to four objectives, SDMA moves further from SFT than DPO. To test whether its advantage persists at comparable KL, we scaled SDMA's trained LoRA update toward SFT until its KL approximately matched DPO's, then generated fresh responses and repeated the DPA--Beaver evaluation. This required no retraining.

ObjectivesPolicyKL (nats/response)MPDLin$_{.05}$Lin.10_{.10}WG.05_{.05}WG_{.10}\dhelpd_{\mathrm{help}}dharmd_{\mathrm{harm}}
2Scaled SDMA22.099.397.166.094.443.5+0.68+0.42
2DPO22.299.890.952.578.831.7+0.69+0.36
3Scaled SDMA31.798.698.784.392.850.2+0.31+0.44
3DPO30.380.090.448.168.33.7+0.16+0.28
4Scaled SDMA30.998.699.693.090.753.5+0.47+0.48
4DPO29.896.899.387.886.743.5+0.52+0.40

Under these judges, SDMA retains a higher MPD at comparable KL with three and four objectives. DPO is slightly higher with two. With five objectives, the original SDMA checkpoint already achieves higher MPD (97.7% versus 94.8%) at lower KL (37.3 versus 44.1 nats per response) as shown in the Table.

Novelty

Our focus is on expected utility guarantees for the broad class of monotonic utility function. The previous work (Farajzadeh et al. 2026) is motivated by performance guarantees for the much narrower class of linear reward functions, which are the basis for classical inverse reinforcement learning (i.e., feature matching methods).

We have updated Theorem 4 to show the bound between the optimal transport cost and the expected regret for the worst-case choice of monotonic utility functions (with rate of change limits). We establish in Corollary 5 that our finite sample estimate of optimal transport cost is an upper bound (in expectation) to the optimal transport cost for the population (i.e., infinite samples). We discuss in Theorem 6 the finite sample approximations from our approach using either entropy regularization (and achieving only nearly stochastic dominance) or exact optimal transport with finite samples approximations that suffer from the ``curse of dimensionality.''

Experimental Details

We add hyperparameter choices and sensitivity (Table 14) along with objective weight and margin choice explanations. We train using mini-batches of 30 different prompt with 3 generated outputs for each (90 total). OT matching is across these prompts.

Reviewer_G6BuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Stochastically Dominant Multi-Objective Alignment (SDMA), a framework for aligning LLMs across multiple objectives without specifying preference trade-offs. SDMA leverages multivariate first-order stochastic dominance and optimal transport-based subdominance minimization to improve a policy relative to a reference policy for all monotonic utility functions. Experiments involving two to five objectives demonstrate more balanced improvements than existing multi-objective alignment methods.

Critical strengths
  1. The paper addresses a problem in multi-objective LLM alignment and provides a well-motivated alternative to scalarization-based approaches.
  2. The proposed framework has a clear theoretical motivation, leveraging multivariate stochastic dominance to accommodate general monotonic utility functions.
  3. The experiments demonstrate substantial improvements over multiple baselines across different numbers of objectives.
Critical weaknesses
  1. The theoretical guarantee requires the optimal transport objective to reach zero. However, the paper does not establish convergence to this condition or provide theoretical guarantees for approximate solutions.
  2. The theoretical and algorithmic contributions are somewhat limited, as the main formulation and optimization procedure largely build upon existing stochastic dominance and optimal transport methods.
  3. The same reward models are used for training and evaluation, raising concerns about reward overoptimization and whether the improvements generalize to actual human preferences.
  4. The experiments are limited to Llama-2-7B and a single dataset. Additionally, computational overhead, hyperparameter sensitivity, and training stability are not sufficiently investigated.
Questions for authors
  1. Can the authors provide theoretical guarantees for approximate stochastic dominance when the optimal transport objective is nonzero?
  2. How does SDMA perform when evaluated using independent reward models or human preference judgments?
  3. How sensitive is SDMA to the hyperparameters α\alpha and β\beta, and how does its computational cost scale with the number of objectives and sampled responses?
  4. Could the authors clarify the main technical novelty relative to the pluralistic stochastic dominance framework of Farajzadeh et al. (2026)?
1 comment
AuthorsOpenReview ↗

Expanded theory, withheld rewards, and other improvements

Thank you for your insightful questions!

Q1 Theory

We replace Theorem 4 with a generalized guarantee for suboptimal optimal transport solutions that focuses on the class of monotonic reward functions (a key technical novelty of our paper) and add finite sample properties (Corollary 5 and Theorem 6).

Theorem 4. The optimal transport cost, OT*, from a finite sample upper bounds the worst possible expected regret of the new policy (with ⊙\odot representing element-wise multiplication): $\mathbb{E}\left[u(\boldsymbol{\alpha}\odot{\bf R}^0+\boldsymbol{\beta})-u(\boldsymbol{\alpha}\odot{\bf R})\right] \leq OT^*({\bf R},{\bf R}^0) $ (where the transport cost is subdominance with parameters α,β\alpha,\beta) for all 1-Lipschitz monotonic utility functions $u : |u({\bf x})-u({\bf y})| \leq ||{\bf x}-{\bf y}||_\infty$, and α>0\boldsymbol{\alpha} > {\bf 0} and β≥0\boldsymbol{\beta} \geq {\bf 0}.

Our proof (Appendix) shows that Kantorovich-Rubinstein duality holds for the α,β\alpha,\beta-translated reward vectors.

Corollary 5. The finite-sample estimator of optimal transport cost has greater expected transport cost compared to the population optimal transport cost. This can be seen from the dual established by Theorem 4:

$\mathbb{E}\left[ \sup_{u \in \mathcal{U}} \frac{1}{m} \sum_i u(\boldsymbol{\alpha}\odot{\bf R}^0_i+\boldsymbol{\beta})-\frac{1}{n} \sum_i u(\boldsymbol{\alpha}\odot{\bf R_i}) \right]\geq \sup_{u \in \mathcal{U}} \mathbb{E}\left[u(\boldsymbol{\alpha}\odot{\bf R}^0+\boldsymbol{\beta}) \right] - \mathbb{E}\left[u(\boldsymbol{\alpha}\odot{\bf R})\right]$.

This inequality is a consequence of Jensen's inequality (i.e., E-sup ≥\geq sup-E).

Theorem 6 (Groppe and Hundrieser (2024)). The deviation of the finite sample estimator of optimal transport cost OT^\widehat{OT} with nn samples and entropy regularization from the infinite sample limit OTOT is: ∣OT∗(R0,R)−OT^(R0,R)∣≲n−1/2|OT^*({\bf R}^0,{\bf R})-\widehat{OT}({\bf R}^0,{\bf R})| \lesssim n^{-1/2} for entropy-regularized OT (and n−1/Kn^{-1/K} for exact OT and KK reward features).

The key takeaway is that the finite sample OT we minimize is a (noisy) expected upper bound on the true optimal transport/amount of stochastic dominance violation.

Q2 Reward Hacking?

Our new evaluations with withheld rewards show our performance generalizes to other withheld rewards. We evaluate SDMA and all seven baselines using DPA for helpfulness and negated Beaver cost for harmlessness (neither used during training). In the two-objective setting, SDMA improves with respect to SFT on both held-out measures (+0.45 helpfulness; +0.77 harmlessness; SDMA and DPO tied at 99.8% matched Pareto dominance: Appendix D, Table 7).

SDMA yields 96.4%, 97.9%, and 97.7% matched Pareto dominance for K=3,4,5K=3,4,5 compared to 80.0%, 96.8%, and 94.8% for DPO (Appendix D, Tables 3, 8-10), which was the strongest baseline at each KK. These results show helpfulness and harmlessness gains transfer to the held-out judges we report.

Q3 Experiment Details

We add hyperparameter choices and sensitivity (Table 14) along with objective weight and margin choice explanations. With two objectives (K=2K=2), we fix the weights uniformly and select the margin using a preliminary sweep on an earlier reference policy. For K=3,4,5K=3,4,5, we initialize weights uniformly and learn them during training. We scale the margin down as the number of objectives increases so each objective faces the same initial improvement requirement. Appendix F gives the sweep values and the settings for each experiment.

Computational cost (new Appendix G, Tables 16–17). We add wall-clock training times for every method and a breakdown of SDMA's runtime across K=2,3,4,5K=2,3,4,5 objectives. Generation and reward scoring account for the most time. Transport matching, including validation, accounts for less than 1% of runtime. Our timing measurements use three responses per prompt.

Training stability and scope (new Appendix E, Tables 10–13). We add results from three independent SDMA training seeds for each KK. Under the original reward models, matched Pareto dominance varies little across seeds with K=2,3,4K=2,3,4 objectives and shows more variation with K=5K=5. These experiments assess stability on Llama-2-7B and HH-RLHF. Generalization to other backbones and datasets remains open.

Q4 Novelty

We focus on expected utility guarantees for the broad class of monotonic utility functions, which is the most general class of utility functions compatible with the ordering of reward features. Farajzadeh et al. (2026) establishes performance guarantees (and a loose convex upper bound) for the much narrower class of linear reward functions, which are the basis for classical inverse reinforcement learning (i.e., feature matching methods). Though the solution algorithms for both works largely converge, we believe addressing monotonic utility functions is an important contribution to LLM alignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.