- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to improve several LLM reward functions simultaneously relative to a fixed reference policy. Its target is multivariate stochastic dominance: improvement for every coordinatewise monotone utility of the joint reward vector. The proposed method, SDMA, uses optimal transport to match samples from the current and reference policies, penalizes reward shortfalls, and converts the matching into a weighted-best-match policy update. Experiments use Llama-2-7B model and HH-RLHF dataset with two to five objectives. Compared with multiple alignment baselines, SDMA reports higher empirical dominance measures and positive improvements across all five reward functions in the main experiment.
Critical strengths
-
A meaningful alignment goal. Definition 2 asks for improvement over a reference policy under every coordinatewise monotone utility. This is a useful goal when several response qualities matter and the user's utility function is not known in advance. It gives the paper a clear motivation for studying the joint reward distribution.
-
The reported gains are substantial. In the five-objective setting, Table 1 gives an MPD of 94.1% for SDMA and 22.1% for the best trained baseline on this metric. SDMA also has positive effect sizes on all five objectives. Table 2 reports strong dominance metrics across two to five objectives. The size and consistency of these gains make the method worth further study.
Critical weaknesses
-
The policy update needs a derivation. Theorems 3 and Theorem 4 give structural results from prior work. The missing step is the connection of Eq. (5) to Algorithm 1, line 4. The paper does not explain why this update can reduce the transport cost. I would like to see the surrogate being optimized and a derivation of the coefficients . There is also a distinction between matching finite samples and establishing dominance between the population reward distributions.
-
The experiments leave the practical advantage uncertain. Work on conflicting objective gradients includes MGDA [1], CAGrad [2] and subsequent approaches for LLM alignment, such as RACO [3] and GAPO [4]. These are highly related to the idea of simultaneous improvement for multiple objectives. Comparing with these methods would help establish how much SDMA gains over approaches that explicitly address gradient conflicts.
-
The paper gives no training time, GPU-hours, or training OT solver, so the overhead of the transport step is difficult to assess. Reward normalization and the numerical settings of and also need to be specified. Finally, the five generation trials in Section 4.3 measure variation in sampled responses; they do not show whether the results are consistent across independent training runs.
References
[1] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique, 350(5–6):313–318, 2012.
[2] Bo Liu et al. Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS, 2021.
[3] Peter Chen et al. Reward-free Alignment for Conflicting Objectives. ICML, 2026.
[4] Chengao Li et al. Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models. ACL, 2025.
Questions for authors
-
Could you derive the weighted-best-match update in Algorithm 1 and identify the objective optimized by the implementation? Please explain how it relates to the transport objective when responses are resampled and the matching changes.
-
Could you compare SDMA with existing approaches that explicitly address gradient conflicts under a fair setup, e.g., with comparable compute budgets?
-
Please report wall-clock training time and the OT solver, including the fraction of runtime spent constructing and solving the OT problems. How does the comparison change under similar total compute? Results from multiple independent training seeds would also help assess the reliability of the reported gains.
-
Please discuss Kaya et al. [5], which also studies policy improvement using stochastic dominance in multi-objective RL. How do the dominance criterion, choice of reference policy, and LLM setting distinguish the contribution here?
References
[5] Ege Can Kaya et al. Stochastic Dominance Driven First-Order Policy Optimization for Multi-Objective Reinforcement Learning. UAI, 2026.
3 comments
Thank you for the clarifications and additional results. My concern in Q3 and Q4 have been resolved.
However, for Q1, the response does not provide the requested derivation connecting the update to the transport objective.
For Q2, I understand that methods [1]-[4] do not target stochastic dominance. However, I suppose your ultimate goal is to improve performance in the multiple objectives context. Stochastic dominance is just one way to pursue this goal, and other ways may be better or worse. A comparison would therefore help understand whether SDMA performs better than these alternatives.
Thus, I would like to keep my score 2.
Thank you for your quick feedback.
[4] is similarly applied to LLMs and its goal (avoid unbalanced optimization of multiple objectives) is most similar to ours. However, it has only been evaluated using two reward dimensions (helpfulness vs. harmlessness). To obtain a diverse frontier (Figure 3), which is required to compete on our evaluation measures, it uses various weightings between the two reward dimensions {0.2, 0.4, 0.6, 0.8} to train separate models. Our largest experiments have five reward dimensions. The approach in [4] does not scale in a reasonable way to provide comparisons that would be competitive against our existing baselines.
Clarifications, positioning, and novelty
Thank you for your clarification questions! We hope our responses help distinguish our approach from existing work.
Q1. Explanation of policy gradient updates.
For each demonstration we sample multiple outputs, and each demonstration is matched to multiple outputs. Our policy gradient updates put more gradient weight on the sampled outputs that are more desirable matches, i.e., (closer to) Pareto-dominant with the paired demonstration. Line 3 of our algorithm downweights the worst matched samples. We find empirically that this accelerates the our algorithm towards a better policy during training. We suspect that like policy gradient update weighting in general (Sutton and Barto, 2018), there are many weights that provide convergence, but vary in how quickly.
Q2. Comparison with gradient conflict methods.
The methods in [1]-[4] seek single points/tradeoffs in the competing objectives. Our approach (and distributional approaches more generally) seek a distribution over outputs that reflect many different tradeoffs across multi-objectives. This is generally required to achieve stochastic dominance because each demonstration may reflect different trade-offs, so a single point/tradeoff will not be flexible enough to achieve stochastic dominance (and provide corresponding monotonic utility guarantees).
Q3. Run Time
Computational cost. The original SDMA runs took 3.2 to 4.3 hours on one NVIDIA H200 NVL GPU. Most of that time was spent generating responses and scoring them with the reward models. At each training step, we use MOSEK to solve a quadratically regularized optimal transport problem matching 30 reference responses to 90 policy responses pooled across prompts. Constructing and solving this problem took approximately 0.06 seconds per step. Including both training and validation matching, optimal transport accounted for less than 1% of wall-clock time. Appendix G reports training times for all methods (Table 16) and the breakdown by phase for SDMA (Table 17).
Comparison at matched training time. For each objective count, we used DPO's training time as the budget and selected the best validation checkpoint reached within that time for each of two independent SDMA training seeds. We evaluated both methods on the test prompts using the original reward models. The table compares mean matched Pareto dominance (MPD, in percent) across the two SDMA seeds with the final DPO checkpoint.
SDMA achieved higher MPD at each matched training budget, but the shorter budget substantially reducedsSDMA's MPD relative to full runs, particularly with 3 and 5 objectives. With 3 objectives, both methods had low MPD. As detailed in Appendix G, these timings exclude one-time data preparation for both methods.
Reward normalization and hyperparameters. We use raw rewards with 2 objectives and standardized rewards using SFT reference statistics with 3 to 5 objectives. The objective weights are fixed at uniform values with 2 objectives and learned from a uniform initialization with 3 to 5. Appendix F (Table 14 and accompanying text) gives the numerical weights and margins, and explains how the margins were chosen.
Training stability. We train SDMA with three independent seeds at each objective count. For each seed, we average five generation trials before computing the mean and standard deviation across training seeds. Under the original reward models, MPD varies little across seeds for 2 to 4 objectives. The five-objective setting showed more variation, with a mean of 89.9% and a standard deviation of 4.3%. Appendix E (Tables 10–13) reports the results for each objective count. Each baseline was trained with 1 seed.
Q4. Discussion of Kaya et al.
Kaya et al. (2026) considers the distributional definition of stochastic dominance based on the cumulative density function (in their paper). Eq. (2) in our paper is the equivalent right-sided version of this (i.e., defined by upper sets). The utility-based definition that guides our optimization (Def. 2) is stronger, as described in lines 194-197, so the Kaya et al. approach does not provide (approximate) performance guarantees for the same broad class of monotonic utility functions that are the focus of our guarantees.
We do not commit to any particular trade-offs in the multiple objectives under our approach and instead require improvement for all monotonic utility functions. This generally requires a stochastic policy that has good coverage of different demonstrations. The LLM setting is challenging (compared to previous work in sequential decision making with two or three main modes of demonstrated behavior) because the reference policy may already produce a distribution over a variety of outputs.