- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Applying smooth Tchebycheff scalarization (STS) directly to rewards introduces two hyperparameters. The paper instead observes that multi-objective RL is itself a multi-objective optimization problem, a vector of KL divergences to optimal policies, and applies STS to that.
The scalarized reward that falls out standardizes each objective by its observed distribution, leaving a single hyperparameter . STOMP plugs this reward into OffsetDPO, and is evaluated by hypervolume on 3 protein and 3 chatbot datasets.
Critical strengths
Scalarizing the optimization problem rather than the rewards turns free offsets and scales into derived quantities. Setting likewise makes each objective's temperature fall out rather than be tuned.
Critical weaknesses
- The experimental setting is weak. For baselines, five of the six are from the DPO family, and three scalarize linearly (DPO-Lin, ODPO-Lin, ERA). ODPO-STZ is the only nonlinear alternative. The authors should try stronger MOO scalarization baselines (e.g., Dynamic, GradNorm, Lagrangian, PAMA, etc), which have already been benchmarked in online RL (https://arxiv.org/abs/2602.06869). Additionally, the Pareto fronts for the protein datasets are built from only three weights, all clustered near the center, which is too coarse and inconsistent with the 9–15 weights used for the chatbot datasets.
- Significant but incremental. The components are STS (Lin et al. 2024), OffsetDPO (Amini et al. 2024), and NLL regularization (Pang et al. 2024). What is new is the derivation that turns the scalarization's free hyperparameters into derived quantities, which I believe is worthwhile but modest for a full paper, given the weak experimental setting noted above.
Questions for authors
Please respond to the weaknesses.
Additional questions:
- A detailed per-objective analysis across the datasets would be essential to demonstrate that STOMP works: do those objectives actually compete, and how does each one improve under the derived parameters?
- The notation and in the analysis is confusing. I assume it refers to the four rows of Figure 1?
Thanks for the clarification. However, I am still not convinced by the test-set explanation. Generating novel sequences does not change the fact that the checkpoints were chosen on the test set. Since the score is noisy, picking the best point also favors methods that fluctuate more, even when every method is treated the same way. I will keep my score.