- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ORPG, which optimizes a separate clipped objective for each reward and merges the two resulting gradients into one update. Compatible gradients are partially rebalanced in magnitude while preserving the norm of their sum. Conflicting gradients are projected, either symmetrically (PCGrad) or so as to protect a primary objective. On Qwen3-4B-Instruct, the authors report gains over GRPO, GDPO, and GD2PO in helpfulness–safety and math correctness–length settings.
Critical strengths
The paper studies a relevant question for multi-reward LLM training that is how per-objective gradients should be combined, given that different objectives might have different priorities. However, as discussed below, the proposed rule and its evaluation do not answer this question convincingly.
Critical weaknesses
Limited novelty and missing key baselines. ORPG’s update reduces to a per-minibatch weighted sum (Eq. 12). Since each gradient is linear in its advantage when clipping is inactive, this essentially reweights the rewards dynamically, as gradient-based dynamic reward weighting does (https://arxiv.org/abs/2509.11452). Yet the paper compares with neither that method nor GAPO. It also omits classic gradient-balancing methods (GradNorm, MGDA, Nash-MTL, FAMO) from MTL literature, which have already been adapted to LLM alignment (https://arxiv.org/abs/2602.06869).
Evaluation too narrow to support the claims. Both settings use only two objectives, and the ORPG's rule is defined only for two. Prior multi-objective LLM work uses at least three objectives for math, and all results here come from one model trained for 100 steps. Even within this setting, the evidence for the core mechanism is weak.
Questions for authors
- Does the method work with three or more objectives? If so, how are the mixing weights and mixed conflict relationships defined?
- Figure 7 shows the compatible rule changes the update direction by under 1. How does such a small change produce the gains attributed to it in Table 4?
- Why did you choose a fixed 4000 token threshold for the length reward instead of comparing each response’s length to a running average, which would keep providing a signal as responses get shorter?