- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies multi-objective alignment of LLMs, where a single model is expected to produce different trade-offs among objectives according to a user-specified preference vector. The paper focuses on the training-time preference distribution: existing preference-conditioned methods typically train under a fixed distribution over the preference simplex, although the difficult regions of the simplex may depend on the objectives and on the current policy.
The proposed method, MAESTRO, conditions a single policy on the preference vector through the prompt and trains it using PPO with linearly scalarized rewards. Its main additional mechanism is an adversarial preference sampler: the sampling distribution is parameterized as a Dirichlet distribution whose concentration parameters are updated online using a score-function estimator and unnormalized exponentiated gradient / online mirror descent, with the goal of allocating more training to poorly served preference regions. A batch-relative rank-based reward surrogate is used for the adversary to reduce sensitivity to heterogeneous reward scales. The method is evaluated on HH-RLHF, BeaverTails, and Reddit TL;DR with two- and three-objective settings. The experiments compare against per-preference training, model-merging approaches, and single-model alignment methods, and include fixed-distribution ablations, multiple-seed Pareto-front metrics, reward-model robustness checks, and training-cost comparisons.
Critical strengths
- The paper identifies a concrete and practically relevant training issue in preference-conditioned MOA and evaluates it with a reasonably well-controlled ablation. The strongest part of the paper is not the general idea of training a steerable multi-objective model, but the observation that a fixed preference-sampling distribution can serve different objective pairs very differently. The fixed-Dirichlet ablation in Fig. 4 is particularly useful because it keeps the policy-training framework largely unchanged while varying the component that MAESTRO is intended to improve. This provides reasonably direct evidence that adaptive allocation of training preferences can improve Pareto coverage and worst-preference performance.
- The empirical evaluation is relatively thorough for the specific mechanism being proposed. Beyond the main Pareto-front plots, the paper reports five-seed HV/IP/MSR/PGAP results, fixed-sampling and surrogate ablations, sensitivity analyses, held-out reward-model evaluation, and a training-cost comparison. In particular, the improvements on the two BeaverTails objective pairs are substantially larger than the reported seed-level variation, which makes the central empirical effect more convincing.
Critical weaknesses
The main weakness is the limited novelty relative to the existing multi-objective alignment literature. Several key ingredients of MAESTRO—single-model preference-conditioned alignment, inference-time preference steering, linear scalarization, and training over sampled preference vectors—have already appeared in closely related works such as Panacea, RiC, CLP and MO-ODPO. The main identifiable novelty therefore appears to be the adaptive adversarial learning of the preference-sampling distribution through the Dirichlet update. While the experiments show this mechanism is useful, I find the overall methodological contribution relatively incremental. I would encourage the authors to clarify more explicitly how MAESTRO differs technically from these closest prior works, and whether the particular coupling of its components yields a genuinely new mechanism or capability beyond a composition of established ingredients.
Questions for authors
-
Appendix E.5 states that the adversary update using Eq. (10) is a stochastic-gradient heuristic rather than an unbiased gradient of the minimax objective in Eq. (6). Which theoretical results in Appendix E should be interpreted as guarantees for the actual Algorithm 1, and which only characterize the idealized minimax objective?
-
The paper argues that the difficult regions of the preference simplex can move as the policy improves. Could the authors show how the learned Dirichlet parameters or preference distribution evolve during training, together with the corresponding per-preference performance? This would help verify whether the adversary is actually tracking dynamically changing hard regions.
5 comments
Q1. Which theoretical results in Appendix E should be interpreted as guarantees for the actual Algorithm 1, and which only characterize the idealized minimax objective?
Appendix E.5 states that the adversary update using Eq. (10) is a stochastic-gradient heuristic rather than an unbiased gradient of the minimax objective in Eq. (6). Which theoretical…
We thank the reviewer for asking us to distinguish these scopes explicitly. Proposition 4 and Corollary 1, added in the revision, describe Algorithm 1, namely what its adversary step estimates and which policy optima it targets, while Propositions 1–3 and Lemma 1 characterize the idealized objective.
As the reviewer notes, the adversary update using Eq. (10) is not an unbiased estimator of the raw-return gradient in Eq. (7), which is why the submission called it a heuristic. Proposition 4 replaces that description: with the policy and the comparison distribution fixed, as the update itself does, the adversary is scored on , the inner objective of Eq. (6) with each reward replaced by its population soft rank, and Eq. (8) with a batch-independent baseline is its score-function gradient estimator for every .
However, since the policy update uses the original scalarized rewards at every sampled preference, the surrogate reaches the policy only through . Using this, Corollary 1 shows that, under exact simultaneous realizability, no choice of the adversary's feedback changes the policy's global maximizers, which solve Eq. (6), so the surrogate changes how training is allocated across preferences without changing the global policy target. Outside exact realizability, our claim about the update is limited to Proposition 4; we do not claim that the alternating procedure converges to a solution of Eq. (6). Empirically, replacing the surrogate with the raw payoff, for which Eq. (8) estimates the gradient (7) up to a constant factor, gives comparable fronts on BeaverTails Harmless–Helpful and slightly worse ones on Humor–Helpful (Figure 6, Appendix D.3).
Propositions 1–3 and Lemma 1 do not involve the adversary's feedback, so they remain valid under their stated assumptions independently of the surrogate; they characterize the robust objective, the ambiguity set, and KL scaling. Our response to Reviewer UyMg's Q1 details Proposition 4 and the scope of each remaining result, and shows that the allocation still follows under Algorithm 1's same-batch baseline (Appendix D.12).
Q2. Could the authors show how the learned Dirichlet parameters or preference distribution evolve during training, together with the corresponding per-preference performance?
The paper argues that the difficult regions of the preference simplex can move as the policy improves. Could the authors… This would help verify whether the adversary is actually tracking dynamically changing hard regions.
We thank the reviewer for this request. Appendix D.12 and Figure 15 report the learned Dirichlet parameters and sampling distributions together with per-preference performance on four settings: HH-RLHF and BeaverTails, each with Harmless–Helpful and Humor–Helpful. We evaluate four checkpoints per setting, 16 checkpoints in total. Difficulty is measured by , the population soft-rank payoff evaluated on held-out prompts with the checkpoint policy frozen, where is the weight on the first objective, so lower places more weight on helpfulness; lower means harder.
The sampling distribution tracks the current difficulty in all four settings. Sampling density is positively correlated with at all 16 checkpoints, with a median correlation of . The concentration on helpfulness rises at every checkpoint, including the last of each run, and the total concentration grows in all four runs, so these trajectories do not support early stabilization at an effectively fixed .
Where the hard region moves, the sampling distribution follows it. On HH-RLHF Humor–Helpful, the hardest preference moves from toward –, and the sampling mean follows from to . In the other three settings, the hardest region remains approximately stable near –. The introduction now points to this observation (Appendix D.12).
Figure 15 presents the sampling trajectories, checkpoint densities, difficulty landscapes, and corresponding fronts together. Our response to Reviewer HipV's Q1 gives the full definition of and discusses how the fronts change.
Dear Reviewer rKJZ,
Thank you again for your review. It raised three concerns: (1) that our contribution is relatively incremental, (2) which results in Appendix E are guarantees for Algorithm 1 (also raised by Reviewer UyMg), and (3) whether the adversary is actually tracking hard regions (also raised by Reviewer HipV).
For (1), as the reviewer noted, the contribution of our work lies in learning the preference-sampling distribution: prior work trains on preferences drawn from a fixed distribution or from relabeled data, whereas MAESTRO makes this distribution the inner variable of a minimax problem. This inner problem is not trivial: the worst-case distribution depends on the current policy and can move during training. We therefore parameterize it as a Dirichlet and update it online from the current policy's rollouts. In Table 8, we extended the fixed- ablation to all five two-objective settings over five seeds, which isolates the mechanism: MAESTRO ranks first in 11 of the 20 setting–metric pairs. For (2), we added Proposition 4 and Corollary 1 (Appendix E.5), which clarify the relation between the objective and Algorithm 1 itself, while Propositions 1–3 and Lemma 1 characterize the idealized objective regardless of the algorithm. For (3), Figure 15 shows that the sampling density is positively correlated with per-preference difficulty at all 16 checkpoints and follows the hard region where it moves (HH-RLHF Humor–Helpful), and we scoped the introduction's claim to this setting.
As the discussion period approaches its end, we would greatly appreciate hearing whether these revisions resolve your concerns, or whether any point remains insufficiently resolved. Please feel free to ask if you have any further questions.
Thank you,
Authors
We thank the reviewer for recognizing the practical importance of preference sampling, for highlighting the controlled fixed-Dirichlet ablation in Figure 4 as evidence that adaptive allocation can improve Pareto coverage, and for noting the breadth of our empirical evaluation and the gains on both BeaverTails pairs.
We appreciate the questions about MAESTRO’s technical novelty, the relationship between Algorithm 1 and the analysis in Appendix E, and the evolution of the preference distribution during training. We address these points below through (i) a comparison with the closest prior work, (ii) Proposition 4 and Corollary 1, newly added in Appendix E.5, and (iii) new trajectory diagnostics alongside per-preference performance in Appendix D.12 and Figure 15.
W1. The main weakness is the limited novelty relative to the existing multi-objective alignment literature.
As the reviewer notes, adaptive preference sampling is the central contribution. MAESTRO formulates the training preference distribution as the inner optimization variable of a minimax alignment problem and updates it online from the current policy's rollouts. Below, we explain what motivated the formulation, what makes its inner minimization challenging and how we address it, and what the experiments show beyond the evaluations in prior work.
What motivated the formulation. Panacea samples training preferences from a fixed distribution over the simplex, while RiC takes them from the relabeled data on which it trains a reward-conditioned policy. CLP and MO-ODPO sample from a fixed Dirichlet and both report that its concentration matters: MO-ODPO tunes it per task, and CLP finds a prompt-conditioned baseline sensitive to it. Our fixed-α ablation extends this comparison to five settings and five seeds (Figure 4, Table 8). This motivates treating the sampling distribution as part of the alignment problem, so we formulate a minimax objective over an ambiguity set P of preference distributions (4).
What makes the inner minimization challenging. However, the worst-case distribution depends on the current policy and can move as the policy is trained, so it has to be tracked during training, and its payoff is observable only through rollouts of that policy. Over all preference distributions, moreover, the minimum is attained at a point mass on the single worst preference, which credits improvement there alone. Finally, since the inner minimization compares scalarized returns across preferences, an objective whose rewards are numerically larger can dominate which preferences it selects.
How we address it. We parameterize the inner distribution as a Dirichlet, which keeps it from collapsing to a point mass, and update its concentration by unnormalized exponentiated gradient (UEG) with a score-function estimate from the current policy's rollouts. Interleaving these updates with PPO lets the allocation of a finite training budget respond to which preferences the policy currently serves poorly. We also score the adversary on the per-objective rank surrogate of Eq. (10), which compares each reward only within its own objective and bounds it to [0,1]; Proposition 4 identifies the objective this update descends (Appendix E.5).
What the experiments show. The fixed-α ablation (Appendix D.13, Table 8) isolates the adversary by holding the policy architecture, rewards, data, and PPO update fixed. MAESTRO ranks first in 11 of the 20 setting–metric combinations and second in 3 more. The best fixed concentration varies by setting and metric, whereas MAESTRO uses the same adversary learning rate (η=0.1) and clamps on α in all five two-objective settings and achieves the best average HV rank (1.4 vs. 2.2–3.4 for the fixed baselines). On BeaverTails, it improves mean HV over the strongest fixed baseline on both pairs (10.590 vs. 10.121 on Harmless–Helpful, 8.867 vs. 8.265 on Humor–Helpful). While this ablation measures the benefit of learning the distribution, Appendix D.12 (Figure 15) shows how the policy-dependent allocation operates within a training run: at all 16 checkpoints across four settings, the sampling density is positively correlated with the preference difficulty −ht.
Beyond the adversary, on both datasets a policy trained on Harmless–Helpful traces a front on Humor–Helpful when we change only the prefix at inference (Figure 13), while the base SFT model given the same prompts produces no comparable front, evidence that the learned preference steering can transfer across the tested objective pairs. When all methods' responses are re-scored with held-out RMs (Figure 8), MAESTRO retains the outermost Harmless–Helpful front on BeaverTails and stays level with MORLHF on HH-RLHF.