- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GPDPO, a multi-reward policy optimization method that builds on GD2PO. GD2PO drops responses whose per-reward advantages disagree in sign during multi-reward GRPO training. The authors show that this filter depends heavily on the group composition, and that most filtered responses can still be clearly ranked against at least one other response in their group (no worse on every reward and better on at least one, or the reverse). GPDPO keeps the filter but adds an auxiliary pairwise loss on these ranked pairs, so filtered responses still get a training signal, without extra sampling or reward computation. The authors use the method to train Qwen3-1.7B and 4B on RLLA-4K under two- and three-reward setups, and evaluate them on API-Bank. GPDPO gets the best overall accuracy in all cases, about 0.8–2.5 points above GD2PO. Ablations and conflict-rate curves are used to support the design choices.
Critical strengths
The method makes better use of rollouts that are already paid for. GD2PO simply throws away conflicting responses, even though generating and scoring them has already cost compute. GPDPO recovers a training signal from these responses through pairwise comparisons, with no extra sampling or reward evaluation, and this gives consistent gains over GD2PO across both model sizes and both reward setups. This is the main reason I see value in the paper: it addresses a real source of waste in conflict-filtering methods, and it is easy to add to existing GRPO-style training pipelines.
Critical weaknesses
- The novelty over existing Pareto-based methods is limited. Using Pareto dominance within GRPO groups is not new. For instance, ParetoPO already uses Pareto ranking to assign advantages in tool-use RL. Building preference pairs from Pareto-consistent comparisons has also been done on the DPO side, e.g., SIPO and Reward Consistency Sampling. GPDPO’s main addition is applying a pairwise loss only to filtered responses. However, the paper does not compare against ParetoPO, so it is unclear whether this approach actually helps over existing Pareto-based methods.
- The experimental evidence is not sufficient. The evaluation of GPDPO is done only on one benchmark, API-Bank (a tool-calling benchmark), and the gains over GD2PO are 0.8–2.5 points, which is only about 5 to 15 questions. With a single run and no seeds or variance, it is hard to tell whether these gains are real. Since the paper’s value depends on the method reliably helping, this is the main factor in my assessment. Including more domains, such as math or coding, would make the claims more convincing.
Questions for authors
NA
3 comments
Response to Reviewer Mtas (2/2)
Q3. Evaluation of GPDPO across Additional Runs and Domains
Following prior work on tool-calling RL (ToolRL, GDPO, and GD²PO), our main API-Bank experiments report greedy evaluation results from one training run per configuration. Because multi-seed training of all methods was not feasible within the rebuttal period, the analyses below quantify evaluation uncertainty (i.e., variation due to decoding randomness) for these fixed checkpoints and do not capture training variability across different seeds. We will report results over multiple training seeds in the final version.
Performance of GPDPO across repeated evaluations. To address your concern about the repeatability of the reported accuracy gains over GD²PO, we re-evaluated the same Qwen3-1.7B checkpoints over 16 rounds with stochastic decoding (temperature = 0.6, top-p = 0.95, top-k disabled). GPDPO achieved the highest mean Correct Acc. and Overall score in both settings. Its mean accuracy exceeded that of GD²PO by 1.36 percentage points in 2R (64.75% vs. 63.39%) and 0.25 percentage points in 3R (63.55% vs. 63.30%), and its gains over GRPO were 3.43 and 2.20 percentage points, respectively. Tables R8 and R9 report the mean and sample standard deviation across the 16 rounds.
Table R8. API-Bank · Qwen3-1.7B (2R)
| Method | Level 1 Acc. (%) | Level 2 Acc. (%) | Level 3 Acc. (%) | Correct Acc. (%) | Length Rew. | Overall |
|---|---|---|---|---|---|---|
| GRPO | 67.23 ± 0.98 | 54.76 ± 2.42 | 46.66 ± 1.60 | 61.32 ± 0.78 | 0.9665 ± 0.0020 | 1.580 ± 0.008 |
| GDPO | 67.09 ± 0.95 | 58.77 ± 2.10 | 42.56 ± 1.54 | 60.77 ± 0.73 | 0.9716 ± 0.0016 | 1.579 ± 0.007 |
| GD²PO | 69.63 ± 0.67 | 61.57 ± 2.00 | 45.32 ± 1.15 | 63.39 ± 0.48 | 0.9806 ± 0.0013 | 1.614 ± 0.005 |
| GPDPO | 70.97 ± 0.94 | 62.97 ± 2.26 | 46.71 ± 0.89 | 64.75 ± 0.62 | 0.9810 ± 0.0014 | 1.628 ± 0.007 |
Table R9. API-Bank · Qwen3-1.7B (3R)
| Method | Level 1 Acc. (%) | Level 2 Acc. (%) | Level 3 Acc. (%) | Correct Acc. (%) | Format Acc. (%) | Length Rew. | Overall |
|---|---|---|---|---|---|---|---|
| GRPO | 67.53 ± 0.79 | 60.45 ± 2.44 | 42.99 ± 1.57 | 61.35 ± 0.71 | 99.82 ± 0.18 | 0.9553 ± 0.0022 | 2.567 ± 0.010 |
| GDPO | 66.51 ± 0.88 | 55.50 ± 2.39 | 43.65 ± 1.51 | 60.26 ± 0.72 | 99.96 ± 0.07 | 0.9580 ± 0.0025 | 2.560 ± 0.007 |
| GD²PO | 69.38 ± 0.76 | 61.57 ± 2.08 | 45.66 ± 1.79 | 63.30 ± 0.85 | 99.97 ± 0.07 | 0.9743 ± 0.0018 | 2.607 ± 0.010 |
| GPDPO | 70.05 ± 0.90 | 62.13 ± 1.96 | 44.47 ± 1.43 | 63.55 ± 0.75 | 99.92 ± 0.11 | 0.9780 ± 0.0014 | 2.613 ± 0.008 |
To examine whether the gains also appear at the question level, we counted how many times GPDPO and GD²PO answered each question correctly across the 16 evaluation rounds in 2R. For each question, we applied a one-sided Fisher’s exact test to these counts to assess whether GPDPO was more likely than GD²PO to answer it correctly (nominal significance level of 0.05). As shown in Table R10, GPDPO achieved significantly higher accuracy than GD²PO on 19 questions, providing question-level evidence consistent with the aggregate gains.
Table R10. Per-question comparison with GD²PO (2R)
| Level | Total questions | Questions favoring GPDPO (p<0.05) |
|---|---|---|
| Level 1 | 399 | 13 |
| Level 2 | 67 | 4 |
| Level 3 | 131 | 2 |
| Total | 597 | 19 |
Performance of GPDPO on mathematical reasoning. Following your suggestion to evaluate GPDPO beyond API-Bank, we adopted the training set and code from ToRL, trained GD²PO and GPDPO from Qwen3-1.7B, and evaluated them on MATH500 with greedy decoding. As shown in Table R11, GPDPO outperformed GD²PO by 1.40 percentage points in exact-match accuracy, indicating that its gains extend to mathematical reasoning.
Table R11. MATH500 · Qwen3-1.7B
| Method | Accuracy (%) |
|---|---|
| GD²PO | 73.00 |
| GPDPO | 74.40 |
Dear Reviewer Mtas,
Thank you for your valuable feedback. In our responses, we addressed your questions about how GPDPO differs from prior Pareto-based methods, how it compares empirically with ParetoPO, and whether the accuracy gains hold across repeated evaluations on API-Bank and extend to mathematical reasoning on MATH500.
As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.
Best regards, The Authors
Response to Reviewer Mtas (1/2)
We thank the reviewer for raising the comparison with prior Pareto-based methods and for suggesting an evaluation beyond API-Bank.
Q1. Novelty over Prior Pareto-Based Methods
Distinction from ParetoPO. The key difference between GPDPO and ParetoPO lies in how Pareto information is incorporated into policy optimization. ParetoPO converts Pareto rankings directly into policy advantages. Each response receives a base advantage determined by its Pareto layer, and this advantage is then adjusted within the layer. However, this within-layer adjustment is bounded, so responses in the same layer may receive similar advantages even when their underlying rewards differ substantially. More importantly, every response receives a positive advantage under this ranking scheme, including responses that are strictly dominated by another response. Even when all responses in a group fail identically, each response still receives a positive advantage, which can introduce misleading policy update signals and reinforce failed responses.
In contrast, GPDPO does not directly convert Pareto rankings into policy advantages. Instead, it keeps the original policy update, with signed, reward-derived advantages for retained responses, and uses Pareto dominance relations to construct an auxiliary pairwise loss for filtered responses involved in valid Pareto pairs. Such pairs are common, as most filtered responses (88.42% in 2R and 86.67% in 3R) still have a clear dominance relation with at least one peer (Figure 2(d)). Pareto comparisons therefore provide an additional source of supervision, recovering learning signals that would otherwise be discarded, rather than replacing the original reward-derived policy advantages with rank-based ones.
In summary, while ParetoPO uses Pareto layers to determine the policy update signal itself, GPDPO applies pairwise learning to specific Pareto dominance relations to recover the supervision that conflict filtering discards. This distinction constitutes the central methodological novelty of GPDPO.
Distinction from DPO-based methods. GPDPO incorporates Pareto supervision into online RL using responses that have already been sampled and scored for the policy-gradient update. In contrast, SIPO and RCS rely on additional sampling to construct data for preference optimization. SIPO samples and refines responses to find one that dominates the original responses, and then anchors all resulting DPO pairs to that selected response. RCS scores an expanded pool of responses and selects one reward-consistent pair per prompt to build an offline preference dataset. GPDPO instead uses every within-group Pareto pair involving a conflict-filtered response to recover supervision from responses already collected for online RL, so it requires no additional sampling or reward evaluation.
Q2. Empirical Comparison with ParetoPO
Following your advice, we adapted ParetoPO, as described in Algorithm 1 of the ParetoPO paper, to the Qwen3-1.7B 2R API-Bank setting, using the same rewards, generation budget, and evaluation protocol as in our main comparison. Since the official repository linked in the paper currently contains only a README and no implementation, we re-implemented ParetoPO as faithfully as possible from the method description and pseudocode provided in the paper.
In this setting, ParetoPO reached 55.78% accuracy on API-Bank, lower than the 64.99% achieved by GPDPO (Table R6). We attribute this performance gap to the property of its rank-based advantage noted in Q1. ParetoPO assigns a +1 advantage to every response when all responses in a group fail identically, whereas GPDPO gives such a group zero group-relative advantage and forms no Pareto pairs from it. In tool calling, reasoning until truncation is a common mode.
Table R6. API-Bank · Qwen3-1.7B (2R)
To further compare the two methods on mathematical reasoning, we trained GPDPO and ParetoPO from Qwen3-1.7B and evaluated them on MATH500. In this setting, GPDPO outperformed it by 2.00 percentage points in exact-match accuracy (Table R7), indicating GPDPO’s effectiveness in mathematical reasoning.
Table R7. MATH500 · Qwen3-1.7B