- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that in multi-reward RL (e.g., GDPO), rewards are combined into one advantage before PPO clipping, so once the ratio is clipped, signals from rewards with the opposite sign are lost. PACPO splits the advantage into positive and negative parts, keeps part of their overlap, and clips the two parts separately. Experiments are on tool calling, scientific QA and summarization, compared with GRPO, GDPO and GD2PO.
Critical strengths
The problem is stated clearly, and the method can be implement with no extra cost.
Critical weaknesses
- Limited novelty. As the paper's own appendix derivation shows, PACPO is the same as GDPO inside the clipping range, and outside it just adds a penalty that pulls the ratio back, scaled by the reward disagreement. This is a small change to the clipping rule, not really a new way to handle multiple rewards. Since all rewards share the same ratio, it cannot give reward-specific update directions as the paper suggests.
- The experiments do not show that the polarity part matters. There is no comparison with a simple version that adds the same kind of penalty without depending on reward disagreement, or with other clipping variants. The gains are modest and no significance analysis is provided.
Questions for authors
- How does PACPO compare with GDPO plus the same penalty using a constant coefficient?
- How often does PACPO actually change the gradient compared to GDPO during training?
2 comments
Response to Reviewer MZG5, Novelty 2/2
PACPO’s core innovation is preserving opposing reward contributions through clipping so that reward disagreement directly determines the boundary correction. We identify clipping-boundary gradient masking and address it through two complementary designs: shared-scale polarity decomposition, which exposes the positive–negative overlap while preserving the GDPO advantage, and controlled conflict retention with polarity-specific clipping, which converts that overlap into a tunable corrective derivative.
Eq. A.11 characterizes this mechanism compactly: identical aggregate advantages can contain different amounts of opposing contribution and therefore receive different PACPO boundary corrections. This information is unavailable from the aggregate advantage alone. Unlike SAPO’s smooth attenuation, PACPO explicitly retains the opposing contribution within a response. Unlike GCR-PPO’s objective-wise gradient resolution, it incorporates the correction directly into the shared surrogate, without separate reward-gradient backward passes or additional model parameters.
Our supplementary experiments support the practical value of this distinction:
| Task / Metric ↑ | Comparison | Baseline | PACPO |
|---|---|---|---|
| API-Bank / Correctness (%) | SAPO adaptation | 54.27 | 63.39 |
| API-Bank / Correctness (%) | Recovery-off | 59.13 | 63.39 |
| CNN/DM / Overall | GCR adaptation | 92.75 | 94.56 |
The Recovery-off decrease connects recovered gradients to task performance, while the SAPO and GCR comparisons support effectiveness beyond alternative update rules. The compact equivalent expression provides an efficient implementation of the identified mechanism and polarity-aware design; it does not remove their methodological distinction.
Response to Reviewer MZG5: Clarifications and Additional Experiments
We thank the reviewer for the constructive feedback and address the concerns below with additional evidence.
W 1. Novelty and distinction from GDPO
Response: Appendix A.3’s equivalent form shows that PACPO conditions its boundary correction on reward disagreement absent from the aggregate advantage. For shared-scale contributions Bk, let A=∑kBk and C=(∑k∣Bk∣−∣A∣)/2. Then
=\Psi_\epsilon(\rho,A)-\lambda C[|\rho-1|-\epsilon]_+.$$ $A$ does not determine the positive–negative overlap $C$. Contributions $(1,0)$ and $(2,-1)$ both yield $A=1$, but $C=0$ and $C=1$, respectively. At $\epsilon=0.2$ and $\rho=1.3$, PACPO’s derivatives with respect to $\rho$ are $0$ and $-\lambda$, whereas GDPO and a constant-coefficient penalty cannot distinguish the two cases. Polarity decomposition thus makes the boundary correction depend on disagreement information absent from $A$, while preserving GDPO’s interior update. We will emphasize this distinction and the equivalent form in the main text. PACPO is not tied to GDPO, it also improves GRPO without changing its advantage. We use $B_k=w_k(r_k-\bar r_k)/(s_R+\epsilon)$, so $\sum_kB_k=A^{\mathrm{GRPO}}$. Rewards share the total-reward standard deviation; only clipping changes. Results on API-Bank/Qwen2.5-3B: |Method|Acc| |---|---:| |GRPO|59.13 $\pm$ 3.12| |GRPO+PACPO|61.98 $\pm$ 3.35| **W 1. Shared ratio and reward-specific effects** **Response:** A shared ratio gives each token a common score-function vector, but the gradient coefficient can depend on reward disagreement: $$\nabla_\theta\Gamma =\frac{\partial\Gamma}{\partial\log\rho} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}).$$ For $A>0$ and $\rho>1+\epsilon$, GDPO’s derivative with respect to $\log\rho$ is zero, while PACPO’s is $-\lambda C\rho$. These coefficients can change the combined gradient across tokens, but do not provide independently controllable reward-specific directions. We will clarify this distinction. **W 2 and Q1. Role of polarity and comparison with a fixed penalty** **Response:** We examine the role of polarity through three controls: a fixed penalty, shuffled conflict coefficients, and removal of strictly recovered gradients. **Fixed penalty.** We replace the entire coefficient $\lambda C$ with a constant $\mu$: $$\Gamma_{\mathrm{fixed}}(\rho) =\Psi_\epsilon(\rho,A)-\mu[|\rho-1|-\epsilon]_+.$$ We set $\mu=\lambda\bar C\approx0.02775$, using the valid-token-weighted mean $C$ from GDPO’s first 50 training steps (seed 42). Both arms use the same ratio cap. Results on API-Bank with Qwen2.5-3B-Instruct: | Method | Mean score (4 seeds) | |---|---:| | GDPO | 60.56 $\pm$ 4.58| | GDPO + Fixed | 60.91\$\pm\$ 3.72 | | PACPO | **63.39** $\pm$ 2.37 | **Shuffle and recovery-off.** Using the same task and model, we obtain: | Variant | Acc. (%) (4 seeds) | |---|---:| | PACPO | **63.39** $\pm$ 2.37 | | Shuffle | 57.96 $\pm$ 3.81 | | Recovery-off | 59.13 $\pm$ 4.13 | Shuffling conflict coefficients within equal-length response groups preserves their distribution but breaks the association with each response's reward disagreement. Consequently, performance declines. Recovery-off removes the policy-gradient contribution only where the same-token GDPO derivative is zero and the PACPO derivative is nonzero, preserving other corrections and auxiliary-loss paths. The **4.26-percentage-point** drop in mean accuracy supports the contribution of these recovered gradients to task performance. Table 5 in paper further supports polarity-specific clipping. **W 2. Comparison with other clipping variants** **Response:** We additionally compare with GDPO+GCR [1] and SAPO [2], which resolve conflicts between objective-wise gradients and smooth the clipping response, respectively. PACPO instead conditions its boundary correction on the positive–negative overlap $C$, which the aggregate advantage $A$ does not determine. | Method | Apibank Correct (%) | Dailymail/CNN mean (%) | |---|---|---| | PACPO | 63.39 | 94.56 | | GDPO+GCR | 54.27 | 92.75 | | SAPO | 49.38 | 59.87 | [1] Munn H, et al. Scalable multi-objective robot reinforcement learning through gradient conflict resolution [2] Gao C, et al. Soft Adaptive Policy Optimization. **Q 2. Frequency of gradient changes** **Response:** Along a complete supplementary PACPO run, we compared PACPO and counterfactual GDPO derivatives on the same tokens at the same policy: | Event | Rate (%) | |---|---:| | CIR | 2.843 | | Gradient change | **2.639** | | Strict recovery | **1.564** | CIR, gradient change, and strict recovery denote opposing contributions outside the clipping range, differing PACPO and GDPO derivatives, and a numerically zero GDPO derivative with a nonzero PACPO derivative, respectively. We also measured activation in all three environments: | Task | CIR (%) | Recovery (%) | |---|---:|---:| | API-Bank | 2.843 | 1.564 | | QA | 15.449 | 10.411 | | DailyMail | 2.306 | 1.557 |