acceptodds
Under review as a conference paper at ICLR 2027

PACPO: Polarity-Aware Contribution Policy Optimization for Multi-Reward RL

Abstract

As the deployment of large language models expands to complex reasoning, tool-use, and interactive applications, post-training increasingly involves multiple behavioral objectives. To jointly optimize these objectives, multi-reward reinforcement learning has become a key paradigm in language model post-training. Existing methods typically preserve reward-wise information during advantage estimation but merge these signals before applying probability-ratio clipping, causing all reward dimensions to share the boundary response determined by the aggregated advantage. When the policy ratio falls outside the clipping interval, the shared clipping response may mask corrective gradients from reward contributions of the opposite polarity. To address this issue, we propose Polarity-Aware Contribution Policy Optimization (PACPO), which separates reward contributions by polarity and applies polarity-specific clipping responses. This design allows corrective signals from contributions of opposite polarity to remain active, rather than being suppressed by clipping after aggregation. By preserving these otherwise masked signals, PACPO mitigates clipping-boundary gradient masking, thereby promoting more balanced joint optimization across multi-reward objectives. Experiments across diverse multi-reward post-training settings show that PACPO consistently outperforms existing multi-reward optimization methods, demonstrating its effectiveness for multi-reward policy optimization.

Primary area:
reinforcement learning
Keywords:
Large language models, Multi-reward reinforcement learning, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MZG5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that in multi-reward RL (e.g., GDPO), rewards are combined into one advantage before PPO clipping, so once the ratio is clipped, signals from rewards with the opposite sign are lost. PACPO splits the advantage into positive and negative parts, keeps part of their overlap, and clips the two parts separately. Experiments are on tool calling, scientific QA and summarization, compared with GRPO, GDPO and GD2PO.

Critical strengths

The problem is stated clearly, and the method can be implement with no extra cost.

Critical weaknesses
  1. Limited novelty. As the paper's own appendix derivation shows, PACPO is the same as GDPO inside the clipping range, and outside it just adds a penalty that pulls the ratio back, scaled by the reward disagreement. This is a small change to the clipping rule, not really a new way to handle multiple rewards. Since all rewards share the same ratio, it cannot give reward-specific update directions as the paper suggests.
  2. The experiments do not show that the polarity part matters. There is no comparison with a simple version that adds the same kind of penalty without depending on reward disagreement, or with other clipping variants. The gains are modest and no significance analysis is provided.
Questions for authors
  1. How does PACPO compare with GDPO plus the same penalty using a constant coefficient?
  2. How often does PACPO actually change the gradient compared to GDPO during training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MZG5: Clarifications and Additional Experiments

We thank the reviewer for the constructive feedback and address the concerns below with additional evidence.

W 1. Novelty and distinction from GDPO

Response: Appendix A.3’s equivalent form shows that PACPO conditions its boundary correction on reward disagreement absent from the aggregate advantage. For shared-scale contributions BkB_k, let A=∑kBkA=\sum_k B_k and C=(∑k∣Bk∣−∣A∣)/2C=(\sum_k|B_k|-|A|)/2. Then

=\Psi_\epsilon(\rho,A)-\lambda C[|\rho-1|-\epsilon]_+.$$ $A$ does not determine the positive–negative overlap $C$. Contributions $(1,0)$ and $(2,-1)$ both yield $A=1$, but $C=0$ and $C=1$, respectively. At $\epsilon=0.2$ and $\rho=1.3$, PACPO’s derivatives with respect to $\rho$ are $0$ and $-\lambda$, whereas GDPO and a constant-coefficient penalty cannot distinguish the two cases. Polarity decomposition thus makes the boundary correction depend on disagreement information absent from $A$, while preserving GDPO’s interior update. We will emphasize this distinction and the equivalent form in the main text. PACPO is not tied to GDPO, it also improves GRPO without changing its advantage. We use $B_k=w_k(r_k-\bar r_k)/(s_R+\epsilon)$, so $\sum_kB_k=A^{\mathrm{GRPO}}$. Rewards share the total-reward standard deviation; only clipping changes. Results on API-Bank/Qwen2.5-3B: |Method|Acc| |---|---:| |GRPO|59.13 $\pm$ 3.12| |GRPO+PACPO|61.98 $\pm$ 3.35| **W 1. Shared ratio and reward-specific effects** **Response:** A shared ratio gives each token a common score-function vector, but the gradient coefficient can depend on reward disagreement: $$\nabla_\theta\Gamma =\frac{\partial\Gamma}{\partial\log\rho} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}).$$ For $A>0$ and $\rho>1+\epsilon$, GDPO’s derivative with respect to $\log\rho$ is zero, while PACPO’s is $-\lambda C\rho$. These coefficients can change the combined gradient across tokens, but do not provide independently controllable reward-specific directions. We will clarify this distinction. **W 2 and Q1. Role of polarity and comparison with a fixed penalty** **Response:** We examine the role of polarity through three controls: a fixed penalty, shuffled conflict coefficients, and removal of strictly recovered gradients. **Fixed penalty.** We replace the entire coefficient $\lambda C$ with a constant $\mu$: $$\Gamma_{\mathrm{fixed}}(\rho) =\Psi_\epsilon(\rho,A)-\mu[|\rho-1|-\epsilon]_+.$$ We set $\mu=\lambda\bar C\approx0.02775$, using the valid-token-weighted mean $C$ from GDPO’s first 50 training steps (seed 42). Both arms use the same ratio cap. Results on API-Bank with Qwen2.5-3B-Instruct: | Method | Mean score (4 seeds) | |---|---:| | GDPO | 60.56 $\pm$ 4.58| | GDPO + Fixed | 60.91\$\pm\$ 3.72 | | PACPO | **63.39** $\pm$ 2.37 | **Shuffle and recovery-off.** Using the same task and model, we obtain: | Variant | Acc. (%) (4 seeds) | |---|---:| | PACPO | **63.39** $\pm$ 2.37 | | Shuffle | 57.96 $\pm$ 3.81 | | Recovery-off | 59.13 $\pm$ 4.13 | Shuffling conflict coefficients within equal-length response groups preserves their distribution but breaks the association with each response's reward disagreement. Consequently, performance declines. Recovery-off removes the policy-gradient contribution only where the same-token GDPO derivative is zero and the PACPO derivative is nonzero, preserving other corrections and auxiliary-loss paths. The **4.26-percentage-point** drop in mean accuracy supports the contribution of these recovered gradients to task performance. Table 5 in paper further supports polarity-specific clipping. **W 2. Comparison with other clipping variants** **Response:** We additionally compare with GDPO+GCR [1] and SAPO [2], which resolve conflicts between objective-wise gradients and smooth the clipping response, respectively. PACPO instead conditions its boundary correction on the positive–negative overlap $C$, which the aggregate advantage $A$ does not determine. | Method | Apibank Correct (%) | Dailymail/CNN mean (%) | |---|---|---| | PACPO | 63.39 | 94.56 | | GDPO+GCR | 54.27 | 92.75 | | SAPO | 49.38 | 59.87 | [1] Munn H, et al. Scalable multi-objective robot reinforcement learning through gradient conflict resolution [2] Gao C, et al. Soft Adaptive Policy Optimization. **Q 2. Frequency of gradient changes** **Response:** Along a complete supplementary PACPO run, we compared PACPO and counterfactual GDPO derivatives on the same tokens at the same policy: | Event | Rate (%) | |---|---:| | CIR | 2.843 | | Gradient change | **2.639** | | Strict recovery | **1.564** | CIR, gradient change, and strict recovery denote opposing contributions outside the clipping range, differing PACPO and GDPO derivatives, and a numerically zero GDPO derivative with a nonzero PACPO derivative, respectively. We also measured activation in all three environments: | Task | CIR (%) | Recovery (%) | |---|---:|---:| | API-Bank | 2.843 | 1.564 | | QA | 15.449 | 10.411 | | DailyMail | 2.306 | 1.557 |
AuthorsOpenReview ↗

Response to Reviewer MZG5, Novelty 2/2

PACPO’s core innovation is preserving opposing reward contributions through clipping so that reward disagreement directly determines the boundary correction. We identify clipping-boundary gradient masking and address it through two complementary designs: shared-scale polarity decomposition, which exposes the positive–negative overlap while preserving the GDPO advantage, and controlled conflict retention with polarity-specific clipping, which converts that overlap into a tunable corrective derivative.

Eq. A.11 characterizes this mechanism compactly: identical aggregate advantages can contain different amounts of opposing contribution and therefore receive different PACPO boundary corrections. This information is unavailable from the aggregate advantage alone. Unlike SAPO’s smooth attenuation, PACPO explicitly retains the opposing contribution within a response. Unlike GCR-PPO’s objective-wise gradient resolution, it incorporates the correction directly into the shared surrogate, without separate reward-gradient backward passes or additional model parameters.

Our supplementary experiments support the practical value of this distinction:

Task / Metric ↑ComparisonBaselinePACPO
API-Bank / Correctness (%)SAPO adaptation54.2763.39
API-Bank / Correctness (%)Recovery-off59.1363.39
CNN/DM / OverallGCR adaptation92.7594.56

The Recovery-off decrease connects recovered gradients to task performance, while the SAPO and GCR comparisons support effectiveness beyond alternative update rules. The compact equivalent expression provides an efficient implementation of the identified mechanism and polarity-aware design; it does not remove their methodological distinction.

Reviewer_i2LmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a problem in multi-reward reinforcement learning: aggregating reward-wise contribution before clipping can vanish opposing corrective signals when the policy ratio crosses a boundary. To address this problem, the authors propose Polarity-Aware Contribution Policy Optimization (PACPO), which separates reward contributions by polarity and applies polarity-specific clipping responses.

Critical strengths
  1. Authors identify clipping-boundary gradient masking, showing that scalarizing reward-wise contributions before clipping can subject distinct reward dimensions to a unified boundary response and suppress corrective gradients.
Critical weaknesses
  1. The core claim needs more numerical support.
  2. The statistical results of experiments are incomplete.
Questions for authors
  1. The literature may not support authors' claim. The motivation behind the proposed approach seems natural and straightforward. However, there are already several existed work. I am wondering the comparison between PACPO with existed work which should be provided with more details and experiment support, otherwise, we can not identify the novelty clearly.
  2. PACPO is built on GDPO, and the paper should provide more evidence to support the claim that the improvements of performances over PACPO instead of GDPO.
3 comments
AuthorsOpenReview ↗

Thank You for Your Feedback: Clarifications and Additional Evidence

Q 1. Novelty and distinction from related work

Response: Thank you for requesting a clearer comparison with existing methods. PACPO addresses clipping-boundary gradient masking: reward-wise normalization preserves distinct reward signals, but aggregating them before clipping can suppress corrective gradients from opposing contributions. PACPO retains these contributions in a controlled manner and clips the two polarities separately while preserving the original aggregate advantage.

This distinction is explicit in Appendix A.3:

=\Psi_\epsilon(\rho,A)-\lambda C[|\rho-1|-\epsilon]_+, \qquad C=\frac{\sum_k|B_k|-|A|}{2}.$$ Here, $A=\sum_kB_k$ and $\Psi_\epsilon$ is the standard PPO surrogate. **The same $A$ can correspond to different $C$**, allowing PACPO to distinguish reward decompositions that aggregate clipping cannot. PACPO is simple yet effective: it preserves the original interior update while using reward disagreement to correct the clipping-boundary response. The distinction from related approaches concerns both the information used and the intervention: |Method|Main intervention| |---|---| |GDPO| Reward-wise advantage normalization| |SAPO| Smooth, temperature-controlled update gating| |GCR-PPO|Priority-based resolution of reward-gradient conflicts| To compare against gradient-conflict resolution and alternative clipping, we evaluated GDPO + GCR and SAPO: |Method|API-Bank|CNN/DailyMail| |---|---:|---:| |GDPO + GCR|54.27|92.75| |SAPO| 49.38 |59.87| |PACPO|**63.39**|**94.56**| To test applicability beyond GDPO, we applied PACPO to GRPO while preserving its original aggregate advantage: |Method|API-Bank Acc.| |---|---:| | GRPO | 59.13 ± 3.12 | | GRPO + PACPO | **61.98 ± 3.35** | The result support PACPO’s applicability beyond GDPO’s advantage formulation. **W 1. Numerical support for the core mechanism** **Response:** We test whether PACPO changes gradients during training and whether its recovered gradients improve task performance. Along a complete supplementary API-Bank trajectory, we compared PACPO and counterfactual GDPO derivatives on the same tokens at the same policy: | Event | Frequency (%) | |---|---:| | CIR | 2.843 | | Changed local derivative | 2.639 | | Strict zero-to-nonzero recovery | 1.564 | CIR denotes opposing contributions outside the clipping range; strict recovery denotes a numerically zero GDPO derivative but a nonzero PACPO derivative. To test gradient recovery and response-specific conflict information, we use: **Recovery-off** removes only policy-gradient contributions where GDPO’s derivative is numerically zero but PACPO’s is nonzero, retaining other corrections and losses; **Shuffle** permutes conflict coefficients among equal-length responses, preserving their distribution but breaking response matching. | Variant | Correctness (%) | |---|---:| | PACPO | **63.39 ± 2.37** | | Recovery-off | 59.13 ± 4.13 | | Shuffle | 57.96 ± 3.81 | PACPO’s higher scores support both the contribution of recovered gradients and the importance of matching corrections to each response’s reward disagreement. To examine activation beyond API-Bank, we also measure CIR and strict recovery across all three tasks: | Task | CIR (%) |Strict recovery (%)| |---|----|----| | API-Bank | 2.843 |1.564| | Scientific QA | 15.449 |10.411| | CNN/DailyMail | 2.306 |1.557| **Q 2. Contribution beyond the underlying GDPO** **Response:** To isolate PACPO’s contribution, we preserve GDPO’s aggregate advantage and modify only clipping. With identical inputs and normalization statistics, $\lambda=0$ recovers GDPO clipping, while $\lambda>0$ introduces the disagreement-conditioned correction. Table 1 compares PACPO directly with GDPO across three backbones, and Table 5 further isolates polarity-specific clipping. The GRPO extension discussed above also supports applicability beyond GDPO’s advantage formulation. To distinguish this correction from a generic penalty, we replace $\lambda C$ with a constant $\mu$: $$\Gamma_{\mathrm{fixed}}(\rho) =\Psi_\epsilon(\rho,A)-\mu[|\rho-1|-\epsilon]_+.$$ We set $\mu\approx0.02775$ from the valid-token-weighted mean of $\lambda C$ over GDPO’s first 50 training updates, keeping it fixed thereafter. On API-Bank/Qwen2.5-3B-Instruct: | Method | Mean score ± std (4 seeds) | |---|----| | GDPO | 60.56 ± 4.58 | | GDPO + Fixed | 60.91 ± 3.72 | | PACPO | **63.39 ± 2.37** | PACPO achieves a higher mean score while retaining the same underlying advantage construction. Together with the polarity-clipping ablation and recovery-off intervention, this supports an additional contribution from PACPO’s disagreement-conditioned clipping. **W 2. Completeness of statistical reporting** **Response:** Table 1 reports means and standard deviations over five tool-calling runs, as specified in Appendix B.1. Our new supplementary fixed-penalty, shuffle, and recovery-off comparisons additionally report 4-seed means and standard deviations.

Thank you for the reply. I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer i2Lm

Thank you for your careful review and your time during the discussion. We would appreciate it if you could reconsider your assessment in light of the clarifications provided in our response. If these have addressed your main concerns, we respectfully ask you to consider raising your score. If any concerns still affect your assessment, please let us know so that we can clarify them further.

Thank you again for your consideration.

Reviewer_bxSUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACPO, a multi-reward policy optimization method designed to address clipping-boundary gradient masking in LLM post-training. By decomposing reward contributions according to their polarity and applying separate clipping operations, PACPO aims to preserve corrective gradients that would otherwise be suppressed. Experiments on tool calling, scientific question answering, and summarization tasks demonstrate improvements over existing multi-reward optimization baselines.

Critical strengths
  1. Research Motivation: Multi-reward optimization is an important issue in LLM post-training. Different rewards may favor different policy updates, but they still need to be optimized within a single model. The paper looks at how reward aggregation interacts with PPO clipping, a problem that is worth investigating.

  2. Presentation and Efficiency: I found the paper easy to follow. The authors explain the motivation clearly, and the derivations help clarify how PACPO differs from standard PPO clipping. The method is also fairly simple to implement on top of GDPO, without adding model parameters or extra training passes.

Critical weaknesses
  1. Limited Novelty: Eq. A.11 in Appendix A.3 shows that the PACPO objective can be written exactly as the original GDPO/PPO surrogate with an additional conflict-weighted penalty:
ΓPACPO(ρ)=Ψϵ(ρ,A)−λChϵ(ρ).\Gamma_{\mathrm{PACPO}}(\rho)=\Psi_\epsilon(\rho,A)-\lambda C h_\epsilon(\rho).

In other words, PACPO can be viewed as penalizing probability ratios outside the clipping range, with the penalty scaled by reward conflict. This is a much simpler interpretation than the formulation in the main text. The authors do not clearly explain how this differs from existing adaptive clipping or trust-region regularization methods. There is also no direct comparison using the equivalent penalty formulation or simpler regularization baselines. I am therefore not convinced that the polarity-aware decomposition offers much beyond another way of expressing the same objective.

  1. Unclear Benefit of the Recovered Gradients: The paper shows that PACPO can preserve gradients that would otherwise be masked by clipping. However, this does not mean that these gradients necessarily help training. Figure 2 mainly examines clipping rates and gradient behavior, but does not establish whether the recovered gradients lead to better optimization directions. More evidence is needed to connect the proposed mechanism to the reported performance gains.
Questions for authors
  1. Reward Conflicts: I am not sure the experiments adequately reflect the reward conflicts that motivate PACPO. In Table 1, Length is almost saturated for every method. Format is also close to its ceiling on some backbones, leaving Correctness as the main source of improvement. How often do the reward signals actually disagree during training? A breakdown of the results by the degree of reward conflict would make the evidence more convincing.

  2. Evaluation on Recent Models: The evaluation only covers Qwen2.5 and Llama 3.x models, none larger than 8B. I would like to see whether the method still works on newer models, such as Qwen3 or Qwen3.5, or at a larger scale. The current results leave this question open.

  3. Comparison with Related Work: The authors compare against GRPO, GDPO, and GD²PO-Hard, but several other methods have studied multi-reward optimization and gradient conflicts. One example is Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolution (GCR-PPO), which also uses per-reward clipping together with gradient conflict resolution. How does PACPO differ from this line of work? I think the paper needs a closer comparison with related approaches, both in the discussion and, where possible, in the experiments.

  4. Sensitivity to λ: The choice of λ seems to matter quite a bit. In Table 4, increasing it from 0.15 to 0.20 causes Format to drop from 99.82 to 95.64, even though Correctness stays the same. Is this behavior specific to API-Bank? The paper uses λ = 0.15 for all tasks, but I would like to see how sensitive the other tasks are to this choice.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bxSU. Thank You for Your Feedback.

We thank the reviewer for the constructive feedback.

W 1. Novelty and distinction from adaptive clipping

Response: Thank you for highlighting Eq. A.11. It shows that PACPO conditions its boundary correction on disagreement absent from the aggregate advantage. For A=∑kBkA=\sum_kB_k and C=(∑k∣Bk∣−∣A∣)/2C=(\sum_k|B_k|-|A|)/2,

=\Psi_\epsilon(\rho,A)-\lambda C[|\rho-1|-\epsilon]_+,$$ **The same $A$ can correspond to different $C$.** GDPO and a constant-coefficient penalty respond identically to such reward decompositions at the same ratio, whereas PACPO adjusts its boundary correction according to their disagreement. PACPO is simple yet effective. To test disagreement conditioning, we replace $\lambda C$ with a constant in the equivalent penalty form: $$\Gamma_{\mathrm{fixed}}(\rho) =\Psi_\epsilon(\rho,A)-\mu[|\rho-1|-\epsilon]_+.$$ We set $\mu\approx0.02775$ using the valid-token-weighted mean of $\lambda C$ over GDPO’s first 50 training updates (seed 42), keeping it fixed thereafter. Both methods use the same native ratio-cap convention. On API-Bank/Qwen2.5-3B-Instruct: | Method | Mean score ± std (4 seeds) | |---|---:| | GDPO | 60.56 ± 4.58 | | GDPO + Fixed | 60.91 ± 3.72 | | PACPO | **63.39 ± 2.37** | The results support conditioning the clipping correction on each response’s reward disagreement, rather than using a constant penalty or preserving only the coefficient distribution. To test whether PACPO depends on GDPO’s advantage formulation, we also apply it to GRPO while preserving the original aggregate advantage. We define $$B_{i,k}=\frac{w_k(r_{i,k}-\bar r_k)}{s_R+\epsilon}, \qquad \sum_k B_{i,k}=A_i^{\mathrm{GRPO}},$$ where all rewards share the total-reward group standard deviation $S_R$. Only the clipping construction changes. On API-Bank/Qwen2.5-3B-Instruct: | Method | Correctness (%) | |---|---:| | GRPO | 59.13 ± 3.12 | | GRPO + PACPO | **61.98 ± 3.35** | The results show that PACPO can also improve GRPO while preserving its original aggregate advantage. **W 2. Utility of the recovered gradients** **Response:** To test utility beyond gradient availability, **Recovery-off** removes only policy-gradient contributions where GDPO’s derivative is numerically zero but PACPO’s is nonzero, retaining other corrections and losses. | Variant | Correctness (4 seeds) | |---|---:| | PACPO | **63.39 ± 2.37** | | Recovery-off | 59.13 ± 4.13 | The 4.26-point drop supports the recovered component’s contribution to task performance. **Q 1. Reward disagreement during training** **Response:** Final evaluation saturation need not imply reward agreement during training. We therefore examine raw rewards and normalized contributions, counting a prompt group as having **raw reward disagreement** if it contains an answer pair ranked oppositely by two rewards. | Training phase | Groups with raw reward disagreement (%) | |---|---:| | Updates 1–70 | **27.26** | | Updates 71–140 | 1.58 | | Updates 141–210 | 1.17 | | Full trajectory | **10.00** | The training rewards also exhibit substantial early variation: | Reward | Early groups with within-group variation (%) | |---|---:| | Correctness | 48.65 | | Format | 17.71 | | Length | 54.46 | Format and Length vary early despite final saturation, while raw reward conflicts decline during training. **Q 2. Evaluation on newer and larger models** **Response:** Thank you for suggesting this extension. We follow the experimental settings of GDPO and GD²PO, comparing methods on the same Qwen2.5 and Llama backbones to ensure controlled comparisons. We acknowledge that these results do not establish performance on newer or larger models and will consider extending the evaluation to these settings in next version. **Q 3. Comparison with gradient-conflict resolution** **Response:** Analysis of GCR-PPO: | Property | GCR-PPO | PACPO | |---|---|---| | Conflict information | Objective-wise parameter gradients | Signed reward contributions | | Intervention | Priority-based gradient projection | Overlap-conditioned boundary correction | | Separate reward-gradient projections | Yes | No | Our novelty claim concerns controlled retention under aggregate clipping, not per-reward clipping itself. We additionally transferred the **GCR gradient-resolution module** to our GDPO-based LLM training pipeline. | Method |Coherence|Consistency|Fluency|Relevance|Conciseness|Overall| |---|----|----|----|----|----|----| | GDPO + GCR |95.20|91.46|95.45|93.34|88.29|92.75| | PACPO |97.40|93.87|95.82|93.66|92.02|94.56| **Q 4. Sensitivity to λ across tasks** **Response:** To test sensitivity beyond API-Bank, we ran 3 $\lambda$ settings with 3 seeds each on CNN/DailyMail. |λ|U-Mean|Conciseness|Overall| |----|----|----|----| |0.10|95.10±0.47|89.30±0.52|93.94±0.47| |0.15|95.64±0.39|90.94±0.49|94.70±0.42| |0.20|95.01±0.84|90.74±1.12|94.15±0.62| These results support using $\lambda=0.15$ as a common setting across the evaluated tasks without task-specific retuning.

Thank you for the detailed response and additional experiments. The Recovery-off ablation and other analyses help address several of my concerns. My main remaining question is about the novelty of PACPO. Given its exact equivalence to a conflict-weighted boundary penalty , the fixed-penalty comparison is helpful but not fully conclusive.

AuthorsOpenReview ↗

Response to Reviewer bxSU. Thank You for Your Thoughtful Feedback on PACPO’s Novelty. 1/2

Response: Thank you for recognizing the additional experiments and clarifying your remaining concern. We would like to highlight PACPO’s novelty through three connected contributions: a mechanistic finding and two complementary methodological designs.

1. Identifying clipping-boundary gradient masking. Reward-wise normalization can preserve distinct reward contributions during advantage estimation, yet aggregating those contributions before nonlinear PPO clipping can suppress an opposing corrective contribution at the boundary. Our analysis establishes why preserving reward information during advantage estimation alone is insufficient, and identifies the conditions under which the aggregate objective becomes flat while a polarity-specific corrective derivative remains available. This connects multi-reward information loss to the clipping operation itself.

2. Shared-scale polarity decomposition. We construct explicit reward-wise contributions on a common scale while preserving the original GDPO advantage, and separate them into positive and negative components. This exposes both the net preference and the canceled positive–negative overlap. The decomposition makes disagreement information unavailable from the aggregate advantage accessible to policy optimization, providing the basis for a conflict-dependent clipping response.

3. Controlled conflict retention and polarity-specific optimization. We introduce a retention mechanism that separates net preference from opposing overlap and controls how much overlap remains explicit before clipping. We then clip the adjusted positive and negative components separately. This design preserves the original aggregate advantage and interior update while retaining a tunable corrective derivative at the boundary. It connects aggregate clipping at zero retention with full polarity-specific clipping at full retention, without additional model parameters or training passes.

Eq. A.11 exactly characterizes how these designs translate retained disagreement into boundary correction. We further support these contributions through targeted experiments: the same-token analysis verifies the predicted boundary effect, while the fixed-penalty, shuffle, and Recovery-off controls provide complementary evidence for disagreement conditioning, coefficient assignment, and the functional contribution of recovered gradients.

Although PACPO admits a simple equivalent expression, its novelty lies in the underlying mechanistic finding and the corresponding polarity-aware optimization design. The compact form provides an efficient realization of these contributions, whose operational significance is supported by the additional experiments. We will emphasize this connection between the mechanistic analysis, methodological design, and experimental evidence in the revised main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.