acceptodds
Under review as a conference paper at ICLR 2027

ORPG: RECONCILING MULTIPLE REWARD OBJECTIVES THROUGH OBJECTIVE-WISE POLICY GRADIENTS

Abstract

Multi-reward policy optimization requires a joint update that reflects both the learning signals and the intended relationships among objectives. We introduce Objective-wise Reconciled Policy Gradient (ORPG), which constructs a separate clipped policy objective for each reward and reconciles the resulting gradients into one policy update. For compatible gradients, a cosine-dependent interpolation coordinates their contributions through a partially normalized reference while preserving the norm of their sum. We characterize this update as the unique solution of a spherical directional compromise. For conflicting gradients, projection follows the task's priorities. We evaluate the same compatible rule in helpfulness–safety alignment and correctness–cost optimization for mathematical reasoning. ORPG substantially improves average Useful and Harmless scores over the strongest external baseline on each axis. In mathematics, it achieves the highest average full-budget accuracy and three-budget hypervolume among the compared methods, with more accurate and shorter responses than the initial policy. Component comparisons and training dynamics show the larger contribution of compatible coordination and a complementary benefit from conflict handling. These results support gradient reconciliation for objectives with equal standing and for objectives with an explicit priority.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Multi-reward policy optimization, Objective-wise policy gradients

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XsUdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ORPG, which optimizes a separate clipped objective for each reward and merges the two resulting gradients into one update. Compatible gradients are partially rebalanced in magnitude while preserving the norm of their sum. Conflicting gradients are projected, either symmetrically (PCGrad) or so as to protect a primary objective. On Qwen3-4B-Instruct, the authors report gains over GRPO, GDPO, and GD2PO in helpfulness–safety and math correctness–length settings.

Critical strengths

The paper studies a relevant question for multi-reward LLM training that is how per-objective gradients should be combined, given that different objectives might have different priorities. However, as discussed below, the proposed rule and its evaluation do not answer this question convincingly.

Critical weaknesses

Limited novelty and missing key baselines. ORPG’s update reduces to a per-minibatch weighted sum ω₁g₁+ω₂g₂ω_₁g_₁ + ω_₂g_₂ (Eq. 12). Since each gradient is linear in its advantage when clipping is inactive, this essentially reweights the rewards dynamically, as gradient-based dynamic reward weighting does (https://arxiv.org/abs/2509.11452). Yet the paper compares with neither that method nor GAPO. It also omits classic gradient-balancing methods (GradNorm, MGDA, Nash-MTL, FAMO) from MTL literature, which have already been adapted to LLM alignment (https://arxiv.org/abs/2602.06869).

Evaluation too narrow to support the claims. Both settings use only two objectives, and the ORPG's rule is defined only for two. Prior multi-objective LLM work uses at least three objectives for math, and all results here come from one model trained for 100 steps. Even within this setting, the evidence for the core mechanism is weak.

Questions for authors
  1. Does the method work with three or more objectives? If so, how are the mixing weights and mixed conflict relationships defined?
  2. Figure 7 shows the compatible rule changes the update direction by under 1. How does such a small change produce the gains attributed to it in Table 4?
  3. Why did you choose a fixed 4000 token threshold for the length reward instead of comparing each response’s length to a running average, which would keep providing a signal as responses get shorter?
Reviewer_fFd6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Objective-wise Reconciled Policy Gradient (ORPG), a multi-reward policy optimization framework that maintains separate clipped policy objectives for individual rewards and reconciles their gradients into a shared update. For compatible gradients, ORPG introduces cosine-dependent coordination with partial norm normalization, while conflicting gradients are handled through symmetric or priority-aware projection. The method is evaluated on helpfulness–safety alignment and mathematical reasoning with correctness–cost objectives using Qwen3-4B. Experiments report improvements over GRPO, GDPO, and GD2PO, with component ablations supporting the contribution of gradient coordination.

Critical strengths
  1. ORPG addresses an important limitation of existing multi-reward optimization methods by coordinating compatible gradients rather than focusing solely on gradient conflicts. Its cosine-dependent, norm-preserving formulation provides a principled approach to balancing reward objectives with different priorities.
  2. ORPG consistently improves helpfulness and harmlessness scores across three datasets and achieves higher full-budget mathematical accuracy than the compared baselines. The component ablations further demonstrate the importance of compatible-gradient coordination, providing meaningful evidence for the proposed method's effectiveness.
Critical weaknesses
  1. The helpfulness–safety evaluation relies on the same reward and cost models used during training. Consequently, the reported improvements may reflect better optimization of these specific reward models rather than genuine improvements in helpfulness and safety. The lack of independent evaluation weakens the paper's main empirical claims.
  2. Although ORPG achieves the highest full-budget mathematical accuracy, its performance at shorter token budgets is less competitive. Moreover, Appendix Table 9 shows that removing compatible coordination yields higher hypervolume than the full method. These results raise questions about whether the proposed gradient reconciliation consistently provides a better accuracy–cost trade-off.
Questions for authors
  1. The helpfulness–safety evaluation appears to rely on the same reward and cost models used during training. Is there any independent evidence supporting improvements in actual helpfulness and safety beyond these reward-model scores? If not, how should the reported gains be interpreted?
  2. ORPG achieves higher full-budget mathematical accuracy but is less competitive at shorter token budgets. Furthermore, Appendix Table 9 shows that removing compatible coordination yields higher hypervolume than the full method. How do the authors reconcile these findings with the claimed superiority in joint accuracy–cost optimization?
Reviewer_koCcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ORPG, which combines separate policy gradients for multiple rewards into one update. It balances their contributions when they agree and removes opposing components when they conflict, either treating objectives equally or prioritizing one. It also provides a geometric interpretation of the compatible-gradient update. Experiments with qwen3-4b evaluate alignment for helpfulness and safety, as well as the tradeoff between mathematical accuracy and response length. The method is shown to improve alignment reward scores and produces more accurate, shorter mathematical responses than the initial model.

Critical strengths
  • The benefit of balancing gradients (even when they agree) is well supported by experiment (Tab 4) as removing this component lowers Useful and Harmless scores more than removing conflict resolution.
  • Experiments cover both equally weighted objectives and settings that prioritize correctness. Compared with the initial model, ORPG increases mathematical accuracy while shortening responses tokens, though gains over simpler alternatives are smaller.
  • The authors provide a clear and mathematically characterized update. ORPG provides an explicit update rule and proves that its compatible gradient update uniquely solves a geometric optimization problem. The potentially can help understanding the method's behavior more clearly.
Critical weaknesses
  • Most mathematical gains seem to appear without reconciliation. Simply summing the separate gradients achieves 66.04% accuracy, close to ORPG's 66.66% (Tab 9). Comparisons that match how correctness and length are rewarded are needed to isolate reconciliation's contribution.
  • Also, the efficiency improvements don't seem to be consistent. Direct summation achieves slightly higher hypervolume than ORPG. And with a limit of 2,048 tokens, ORPG is less accurate than all external training baselines. These results may suggest an accuracy vs. length tradeoff rather than consistently better efficiency.
  • It would be good to have independent validation for the alignment gains. Currently, the same RMs guide training and evaluate helpfulness and safety. Human assessment or independent evaluators are needed to determine whether higher scores reflect better behavior.
Questions for authors
  1. Could you clarify, if the external mathematical baselines receive the same correct-subset conditioning of the length signal as the proposed ORPG? Please describe how each baseline computes advantages and whether ORPG's accuracy gains persist when compared with training for correctness alone and varying reward weights to achieve similar response lengths.
  2. Do the gains persist under an independent evaluator or blinded human assessment of both helpfulness and harmful compliance? I think this would strengthen the interpretation of Tab 1 beyond improved training reward scores.
  3. How much additional training time and memory does ORPG require compared with GRPO and direct gradient summation under the same hardware and training settings? A relative comparison would help assess its practical benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.