acceptodds
Under review as a conference paper at ICLR 2027

Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarization

Abstract

Large language models can be aligned with human preferences through offline reinforcement learning (RL) on small labeled datasets. While single-objective alignment is well-studied, many real-world applications demand the simultaneous optimization of multiple conflicting rewards, e.g. activity and specificity for proteins, or helpfulness and harmlessness for chatbots. Prior work has largely relied on linear reward scalarization, which provably fails to recover non-convex regions of the Pareto front. In this paper, instead of scalarizing the rewards directly, we frame multi-objective RL itself as an optimization problem to be scalarized via smooth Tchebycheff scalarization, a recent technique that overcomes the drawbacks of linear scalarization. We use this formulation to derive Smooth Tchebycheff Optimization of Multi-Objective Preferences (STOMP), a novel offline RL algorithm that extends direct preference optimization to the multi-objective setting in a principled way by standardizing the individual rewards based on their observed distributions. We empirically validate STOMP using multiple base models on 3 protein engineering and 3 chatbot alignment datasets. Compared to state-of-the-art baselines, STOMP achieves or ties for the highest hypervolumes on 16/18 protein tasks and 5/6 natural language tasks. We thus demonstrate that STOMP is a powerful, robust multi-objective alignment algorithm that can meaningfully improve post-training in multiple domains.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
multi-objective, reinforcement learning, offline reinforcement learning, protein, protein optimization, alignment, dpo

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zZaMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Applying smooth Tchebycheff scalarization (STS) directly to rewards introduces two hyperparameters. The paper instead observes that multi-objective RL is itself a multi-objective optimization problem, a vector of KL divergences to kk optimal policies, and applies STS to that.

The scalarized reward that falls out standardizes each objective by its observed distribution, ρi(x,y)=ri(x,y)σi−γlog⁡Z^i(x),\rho_i(x,y) = \frac{r_i(x,y)}{\sigma_i} - \gamma \log \hat{Z}_i(x), leaving a single hyperparameter γ\gamma. STOMP plugs this reward into OffsetDPO, and is evaluated by hypervolume on 3 protein and 3 chatbot datasets.

Critical strengths

Scalarizing the optimization problem rather than the rewards turns free offsets ιi\iota_i and scales κi\kappa_i into derived quantities. Setting η=1/(2γ2)\eta = 1/(2\gamma^2) likewise makes each objective's temperature βi=γσi\beta_i = \gamma\sigma_i fall out rather than be tuned.

Critical weaknesses
  • The experimental setting is weak. For baselines, five of the six are from the DPO family, and three scalarize linearly (DPO-Lin, ODPO-Lin, ERA). ODPO-STZ is the only nonlinear alternative. The authors should try stronger MOO scalarization baselines (e.g., Dynamic, GradNorm, Lagrangian, PAMA, etc), which have already been benchmarked in online RL (https://arxiv.org/abs/2602.06869). Additionally, the Pareto fronts for the protein datasets are built from only three weights, all clustered near the center, which is too coarse and inconsistent with the 9–15 weights used for the chatbot datasets.
  • Significant but incremental. The components are STS (Lin et al. 2024), OffsetDPO (Amini et al. 2024), and NLL regularization (Pang et al. 2024). What is new is the derivation that turns the scalarization's free hyperparameters into derived quantities, which I believe is worthwhile but modest for a full paper, given the weak experimental setting noted above.
Questions for authors

Please respond to the weaknesses.

Additional questions:

  • A detailed per-objective analysis across the datasets would be essential to demonstrate that STOMP works: do those objectives actually compete, and how does each one improve under the derived parameters?
  • The notation r1−2r_{1-2} and r3−4r_{3-4} in the analysis is confusing. I assume it refers to the four rows of Figure 1?
Reviewer_c7tZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STOMP, an offline multi-objective alignment method. Instead of applying smooth Tchebycheff scalarization (STS) to the rewards directly, the authors apply it to the KL divergences between the policy model and each single-objective optimal policy model. After two relaxations, the objective becomes a scalarized reward in which each reward is divided by its standard deviation and centered at the soft maximum of the same prompt (γ log Ẑ_i(x)). The scalarized reward is used to build preference pairs and as the margin in an OffsetDPO loss. Experiments show that STOMP has the highest or tied-highest hypervolume in 16/18 protein settings and 5/6 chatbot settings.

Critical strengths
  1. The reward centering comes from the derivation, and there is only one single hyperparameter. When STS is applied to the KL divergences instead of the rewards, the term γ log Ẑ_i(x) comes out of the math, and it makes every term ≤ 0. Qiu et al. (2024) need to tune an offset and a scale for each objective. STOMP only needs γ. This makes Tchebycheff scalarization easier to use.

  2. Good results in two very different domains. STOMP has the highest or tied-highest hypervolume in 16/18 protein settings and 5/6 chatbot settings, with three seeds for each setting. The method works for both protein language models and chat models.

Critical weaknesses
  1. No experiment checks the Pareto claim. No experiment shows that STOMP finds the trade-off points that linear scalarization cannot find, e.g., on a task with a known non-convex front.

  2. It is not clear where the gain comes from. The main difference from ODPO-STZ is the centering term (γ log Ẑ_i(x) vs. the mean), but no ablation changes only this term.

  3. The protein results use the test set to pick models. Checkpoints are chosen by test-set hypervolume, and the default τ, γ are also the best on the test set (Tables 4–5), so the numbers may be too high. Also, α-amylase with PG3 is 27.8 in Table 1 but 30.30 in Tables 4–5 with the same setting.

Questions for authors
  1. A task with a known non-convex front would help, to check whether STOMP finds trade-off points that linear scalarization and ODPO-STZ miss.
  2. It would be better to have an ablation that changes only the centering term.
  3. Do the protein results hold if checkpoints, τ, and γ are chosen on a validation split? Why do Table 1 and Tables 4–5 differ for α-amylase with PG3?
  4. Pairs are built with Eq. 9, but the margin uses Eq. 8. What does the final loss optimize?
  5. Section 4.3 uses the same reward model for harmlessness and helpfulness on AnthropicHH. Is this a typo? Should "three-objective SafeRLHF" in Appendix B.2 be HelpSteer2?
  6. STOMP trains one model per preference vector (594 for the chatbot tasks), which is costly when users need many preferences. It would be interesting to apply the STOMP reward to single-model methods such as Rewards-in-Context or Panacea.
1 comment

Thanks for the clarification. However, I am still not convinced by the test-set explanation. Generating novel sequences does not change the fact that the checkpoints were chosen on the test set. Since the score is noisy, picking the best point also favors methods that fluctuate more, even when every method is treated the same way. I will keep my score.

Reviewer_GMj8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

STOMP is an offline preference optimization method for multiple reward objectives. The paper applies smooth Tchebycheff scalarization to the KL divergences between the learned policy and individual reward-optimal policies. This leads to distribution-relative rewards based on standard deviations and estimated partition functions. A fixed reward determines the preference pairs, while a policy-dependent reward enters an OffsetDPO-style loss. Winner likelihood regularization is included where needed. The contribution is a useful, incremental combination of existing ideas into an offline training method. The empirical results are promising: across protein design and chatbot alignment, STOMP achieves or ties for the best hypervolume in 16 of 18 protein evaluation settings and five of six chatbot settings.

Critical strengths

STOMP provides a practical way to bring smooth Tchebycheff scalarization into offline preference training. The use of fixed preference pairs together with a policy-dependent offset is a concrete methodological contribution. Starting from policy-level objectives gives these choices a coherent motivation, even though the derivation needs the qualifications discussed below.

The empirical coverage is good. The paper tests different model families, conflicting objectives, and both two- and three-objective problems, with multiple seeds and preference weights. Matching KL drift in the chatbot experiments is particularly helpful. DPO with linear scalarization is also a strong baseline here, so STOMP's competitive results are meaningful.

Critical weaknesses

Section 2.1 cites Faury et al. [1], Lemma 5, for an exact expression for the optimal multiplier. However, the cited lemma gives an approximation derived using a second-order Taylor expansion. The paper should distinguish this approximation from an exact relationship and clarify its implications for the subsequent derivation. This does not by itself invalidate the proposed training method.

The paper does not report quantitative, matched STOMP comparisons that isolate the effect of using the dynamic rather than the static reward in the loss, or of including the reward-difference offset. The reported qualitative observations do not establish the contribution of these components. The overall results are promising, but these controlled comparisons would make the explanation for the gains more convincing.

[1] Louis Faury, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. Distributionally Robust Counterfactual Risk Minimization. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3850–3857, 2020.

Questions for authors

The method is trained on offline preference pairs. Could you explain the connection between the multi-objective RL derivation and the final preference-learning objective, particularly how the derivation determines the scalarized reward and the dynamic offset term?

Are matched STOMP comparisons available that change only the normalization, the static versus dynamic loss reward, or the reward offset? These would help determine which components account for the improvements.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to the questions and concerns raised in the reviews. Please review their responses and provide any further comments or clarifications, if applicable.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.