acceptodds
Under review as a conference paper at ICLR 2027

GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

Abstract

Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Reward Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MtasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GPDPO, a multi-reward policy optimization method that builds on GD2PO. GD2PO drops responses whose per-reward advantages disagree in sign during multi-reward GRPO training. The authors show that this filter depends heavily on the group composition, and that most filtered responses can still be clearly ranked against at least one other response in their group (no worse on every reward and better on at least one, or the reverse). GPDPO keeps the filter but adds an auxiliary pairwise loss on these ranked pairs, so filtered responses still get a training signal, without extra sampling or reward computation. The authors use the method to train Qwen3-1.7B and 4B on RLLA-4K under two- and three-reward setups, and evaluate them on API-Bank. GPDPO gets the best overall accuracy in all cases, about 0.8–2.5 points above GD2PO. Ablations and conflict-rate curves are used to support the design choices.

Critical strengths

The method makes better use of rollouts that are already paid for. GD2PO simply throws away conflicting responses, even though generating and scoring them has already cost compute. GPDPO recovers a training signal from these responses through pairwise comparisons, with no extra sampling or reward evaluation, and this gives consistent gains over GD2PO across both model sizes and both reward setups. This is the main reason I see value in the paper: it addresses a real source of waste in conflict-filtering methods, and it is easy to add to existing GRPO-style training pipelines.

Critical weaknesses
  1. The novelty over existing Pareto-based methods is limited. Using Pareto dominance within GRPO groups is not new. For instance, ParetoPO already uses Pareto ranking to assign advantages in tool-use RL. Building preference pairs from Pareto-consistent comparisons has also been done on the DPO side, e.g., SIPO and Reward Consistency Sampling. GPDPO’s main addition is applying a pairwise loss only to filtered responses. However, the paper does not compare against ParetoPO, so it is unclear whether this approach actually helps over existing Pareto-based methods.
  2. The experimental evidence is not sufficient. The evaluation of GPDPO is done only on one benchmark, API-Bank (a tool-calling benchmark), and the gains over GD2PO are 0.8–2.5 points, which is only about 5 to 15 questions. With a single run and no seeds or variance, it is hard to tell whether these gains are real. Since the paper’s value depends on the method reliably helping, this is the main factor in my assessment. Including more domains, such as math or coding, would make the claims more convincing.
Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mtas (1/2)

We thank the reviewer for raising the comparison with prior Pareto-based methods and for suggesting an evaluation beyond API-Bank.

Q1. Novelty over Prior Pareto-Based Methods

Distinction from ParetoPO. The key difference between GPDPO and ParetoPO lies in how Pareto information is incorporated into policy optimization. ParetoPO converts Pareto rankings directly into policy advantages. Each response receives a base advantage determined by its Pareto layer, and this advantage is then adjusted within the layer. However, this within-layer adjustment is bounded, so responses in the same layer may receive similar advantages even when their underlying rewards differ substantially. More importantly, every response receives a positive advantage under this ranking scheme, including responses that are strictly dominated by another response. Even when all responses in a group fail identically, each response still receives a positive advantage, which can introduce misleading policy update signals and reinforce failed responses.

In contrast, GPDPO does not directly convert Pareto rankings into policy advantages. Instead, it keeps the original policy update, with signed, reward-derived advantages for retained responses, and uses Pareto dominance relations to construct an auxiliary pairwise loss for filtered responses involved in valid Pareto pairs. Such pairs are common, as most filtered responses (88.42% in 2R and 86.67% in 3R) still have a clear dominance relation with at least one peer (Figure 2(d)). Pareto comparisons therefore provide an additional source of supervision, recovering learning signals that would otherwise be discarded, rather than replacing the original reward-derived policy advantages with rank-based ones.

In summary, while ParetoPO uses Pareto layers to determine the policy update signal itself, GPDPO applies pairwise learning to specific Pareto dominance relations to recover the supervision that conflict filtering discards. This distinction constitutes the central methodological novelty of GPDPO.

Distinction from DPO-based methods. GPDPO incorporates Pareto supervision into online RL using responses that have already been sampled and scored for the policy-gradient update. In contrast, SIPO and RCS rely on additional sampling to construct data for preference optimization. SIPO samples and refines responses to find one that dominates the original responses, and then anchors all resulting DPO pairs to that selected response. RCS scores an expanded pool of responses and selects one reward-consistent pair per prompt to build an offline preference dataset. GPDPO instead uses every within-group Pareto pair involving a conflict-filtered response to recover supervision from responses already collected for online RL, so it requires no additional sampling or reward evaluation.

Q2. Empirical Comparison with ParetoPO

Following your advice, we adapted ParetoPO, as described in Algorithm 1 of the ParetoPO paper, to the Qwen3-1.7B 2R API-Bank setting, using the same rewards, generation budget, and evaluation protocol as in our main comparison. Since the official repository linked in the paper currently contains only a README and no implementation, we re-implemented ParetoPO as faithfully as possible from the method description and pseudocode provided in the paper.

In this setting, ParetoPO reached 55.78% accuracy on API-Bank, lower than the 64.99% achieved by GPDPO (Table R6). We attribute this performance gap to the property of its rank-based advantage noted in Q1. ParetoPO assigns a +1 advantage to every response when all responses in a group fail identically, whereas GPDPO gives such a group zero group-relative advantage and forms no Pareto pairs from it. In tool calling, reasoning until truncation is a common mode.

Table R6. API-Bank · Qwen3-1.7B (2R)

MethodLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Length Rew.Overall
ParetoPO63.6646.2736.6455.780.58581.144
GPDPO70.9361.1948.8564.990.98121.631

To further compare the two methods on mathematical reasoning, we trained GPDPO and ParetoPO from Qwen3-1.7B and evaluated them on MATH500. In this setting, GPDPO outperformed it by 2.00 percentage points in exact-match accuracy (Table R7), indicating GPDPO’s effectiveness in mathematical reasoning.

Table R7. MATH500 · Qwen3-1.7B

MethodAccuracy (%)
ParetoPO72.40
GPDPO74.40
AuthorsOpenReview ↗

Response to Reviewer Mtas (2/2)

Q3. Evaluation of GPDPO across Additional Runs and Domains

Following prior work on tool-calling RL (ToolRL, GDPO, and GD²PO), our main API-Bank experiments report greedy evaluation results from one training run per configuration. Because multi-seed training of all methods was not feasible within the rebuttal period, the analyses below quantify evaluation uncertainty (i.e., variation due to decoding randomness) for these fixed checkpoints and do not capture training variability across different seeds. We will report results over multiple training seeds in the final version.

Performance of GPDPO across repeated evaluations. To address your concern about the repeatability of the reported accuracy gains over GD²PO, we re-evaluated the same Qwen3-1.7B checkpoints over 16 rounds with stochastic decoding (temperature = 0.6, top-p = 0.95, top-k disabled). GPDPO achieved the highest mean Correct Acc. and Overall score in both settings. Its mean accuracy exceeded that of GD²PO by 1.36 percentage points in 2R (64.75% vs. 63.39%) and 0.25 percentage points in 3R (63.55% vs. 63.30%), and its gains over GRPO were 3.43 and 2.20 percentage points, respectively. Tables R8 and R9 report the mean and sample standard deviation across the 16 rounds.

Table R8. API-Bank · Qwen3-1.7B (2R)

MethodLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Length Rew.Overall
GRPO67.23 ± 0.9854.76 ± 2.4246.66 ± 1.6061.32 ± 0.780.9665 ± 0.00201.580 ± 0.008
GDPO67.09 ± 0.9558.77 ± 2.1042.56 ± 1.5460.77 ± 0.730.9716 ± 0.00161.579 ± 0.007
GD²PO69.63 ± 0.6761.57 ± 2.0045.32 ± 1.1563.39 ± 0.480.9806 ± 0.00131.614 ± 0.005
GPDPO70.97 ± 0.9462.97 ± 2.2646.71 ± 0.8964.75 ± 0.620.9810 ± 0.00141.628 ± 0.007

Table R9. API-Bank · Qwen3-1.7B (3R)

MethodLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Format Acc. (%)Length Rew.Overall
GRPO67.53 ± 0.7960.45 ± 2.4442.99 ± 1.5761.35 ± 0.7199.82 ± 0.180.9553 ± 0.00222.567 ± 0.010
GDPO66.51 ± 0.8855.50 ± 2.3943.65 ± 1.5160.26 ± 0.7299.96 ± 0.070.9580 ± 0.00252.560 ± 0.007
GD²PO69.38 ± 0.7661.57 ± 2.0845.66 ± 1.7963.30 ± 0.8599.97 ± 0.070.9743 ± 0.00182.607 ± 0.010
GPDPO70.05 ± 0.9062.13 ± 1.9644.47 ± 1.4363.55 ± 0.7599.92 ± 0.110.9780 ± 0.00142.613 ± 0.008

To examine whether the gains also appear at the question level, we counted how many times GPDPO and GD²PO answered each question correctly across the 16 evaluation rounds in 2R. For each question, we applied a one-sided Fisher’s exact test to these counts to assess whether GPDPO was more likely than GD²PO to answer it correctly (nominal significance level of 0.05). As shown in Table R10, GPDPO achieved significantly higher accuracy than GD²PO on 19 questions, providing question-level evidence consistent with the aggregate gains.

Table R10. Per-question comparison with GD²PO (2R)

LevelTotal questionsQuestions favoring GPDPO (p<0.05)
Level 139913
Level 2674
Level 31312
Total59719

Performance of GPDPO on mathematical reasoning. Following your suggestion to evaluate GPDPO beyond API-Bank, we adopted the training set and code from ToRL, trained GD²PO and GPDPO from Qwen3-1.7B, and evaluated them on MATH500 with greedy decoding. As shown in Table R11, GPDPO outperformed GD²PO by 1.40 percentage points in exact-match accuracy, indicating that its gains extend to mathematical reasoning.

Table R11. MATH500 · Qwen3-1.7B

MethodAccuracy (%)
GD²PO73.00
GPDPO74.40

Dear Reviewer Mtas,

Thank you for your valuable feedback. In our responses, we addressed your questions about how GPDPO differs from prior Pareto-based methods, how it compares empirically with ParetoPO, and whether the accuracy gains hold across repeated evaluations on API-Bank and extend to mathematical reasoning on MATH500.

As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.

Best regards, The Authors

Reviewer_rFvaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GPDPO recovers supervision from responses discarded by multi-reward conflict filtering. A response can be above the group average on one reward and below it on another, yet still have a consistent Pareto comparison with a peer. The method adds a preference loss for pairs involving filtered responses and allows auxiliary gradients only through those responses. The original policy loss handles retained responses. Experiments evaluate two Qwen3 sizes and two reward configurations on API-Bank.

Critical strengths

S1. The analysis motivates a useful use of existing rollouts. Figure 2 shows that many filtered responses retain consistent comparisons with their peers. GPDPO converts these comparisons into updates without additional generation or reward calls. This can recover a signal even when aggregation of the normalized advantages cancels it. The contribution is a useful integration of Pareto supervision with conflict filtering.

S2. Task correctness improves in all four reported settings. GPDPO has the highest correctness in Tables 1 and 2. For Qwen3-1.7B with two rewards, it improves over GD2PO from 62.48% to 64.99%. Both Pareto-based update variants in Table 3 improve over GD2PO, which supports the recovery idea. The loss-weight sweep also gives consistent gains over that baseline. Some secondary rewards decrease. The results establish correctness gains and a better aggregate score, while allowing tradeoffs between rewards.

Critical weaknesses

W1. The ablation does not fully explain the value of the chosen update. Table 3 compares the preference loss with Pareto-guided advantage adjustment, but the alternative advantage formula, normalization, and masking order are not specified. These details matter. At reference refresh, the preference loss assigns filtered responses a signal based on wins minus losses, scaled by response length. This can also be expressed as an advantage update. The methods differ after the policy moves. Without the comparator's definition, the reported gap does not identify the benefit of optimizing preference margins. This limits the component explanation, not the measured benefit of the full method.

Questions for authors

Q1. Please give the advantage formula, normalization, and masking order for the Pareto-guided advantage-adjustment baseline. Does it match GPDPO's initial auxiliary gradient at reference refresh, including length and loss scaling? This would clarify what Table 3 compares. The existing implementation is sufficient; no new training is requested.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rFva

We thank the reviewer for the constructive feedback.

Q1. Clarification of Pareto-Guided Advantage Adjustment and Initial Auxiliary Gradient Matching

Clarification of Pareto-guided advantage adjustment. This ablation uses the same Pareto pairs as GPDPO to assign token-level advantages to filtered responses.

  1. Before masking the base advantages, we identify eligible Pareto pairs from the per-reward group-normalized advantages. A pair is eligible if both responses belong to the same prompt group, one Pareto-dominates the other, and at least one is marked for filtering. Let NEN_E be the number of eligible pairs in the rollout batch.
  2. We take the weighted sum of the per-reward advantages and scale it by each group's retained fraction. We then mask the filtered responses and whiten the base advantages over valid tokens of retained responses.
  3. After whitening, if NE>0N_E > 0, we assign the following adjusted advantage only to valid tokens tt of filtered responses yny_n.

An,tadj=λβ2⏟0.005dnNET^∣yn∣,λ=β=0.1.A_{n,t}^{\mathrm{adj}}=\underbrace{\frac{\lambda \beta}{2}}_{0.005}\frac{d_n}{N_{E}}\frac{\widehat{T}}{|y_n|},\qquad \lambda =\beta =0.1.

dnd_n is the number of these pairs that yny_n wins minus the number it loses. We write ∣yn∣|y_n| for the number of valid tokens in yny_n and T^\widehat{T} for the total number of valid response tokens in the rollout batch divided by four, the number of PPO mini-batches. A filtered response keeps a zero advantage if dn=0d_n=0 or if the batch contains no eligible pairs. The adjusted advantages enter the standard clipped PPO loss directly alongside the base advantages of retained responses.

Initial auxiliary gradient matching. Yes. At reference refresh, the two auxiliary gradients match, including length and loss scaling, up to a mini-batch normalization factor. At refresh, θ=θold\theta=\theta_{\mathrm{old}}, so uθ(yn∣x)=0u_\theta(y_n\mid x)=0 and each pairwise margin is zero. The derivative of the pairwise loss with respect to the margin is therefore −βσ(0)=−β/2-\beta\sigma(0)=-\beta/2. The PPO importance ratio is 11, so clipping is inactive.

\frac{\partial (\lambda L_{\mathrm{pair}})}{\partial \ell^\theta_{n,t}} &=-\frac{\lambda \beta}{2}\frac{d_n}{N_{E}|y_n|}, \\\\ \frac{\partial L_{\mathrm{PPO,adj}}}{\partial \ell^\theta_{n,t}} &=-\frac{A_{n,t}^{\mathrm{adj}}}{T_{k}}=-\frac{\lambda \beta}{2}\frac{d_n}{N_{E}|y_n|}\frac{\widehat{T}}{T_{k}}. \end{aligned}$$ Here $\ell^\theta_{n,t}$ is the current policy's log probability of token $t$ in response $y_n$, and $T_{k}$ is the actual number of valid tokens in the $k$-th PPO mini-batch. $L_{\mathrm{pair}}$ denotes the pairwise loss, and $L_{\mathrm{PPO,adj}}$ is the adjusted-advantage contribution to the PPO loss. All other symbols are as defined above. For filtered responses, the two gradients therefore match up to the factor $\widehat{T}/T_k$, and coincide exactly when all PPO mini-batches contain the same number of valid tokens. In Table 3, advantage adjustment improves Correct Acc. from 62.48% (GD²PO) to 63.82%, indicating that the recovered Pareto relations themselves contain useful optimization signals. Because the two variants produce the same auxiliary gradient at each reference refresh, up to the mini-batch factor above, this comparison holds the strength of the recovered supervision fixed. The further improvement of full GPDPO to 64.99% therefore suggests that directly optimizing pairwise preference margins provides a more effective way to exploit the recovered relational supervision. We will add this formula and derivation to the appendix.

Thanks for clarifying the formula for the adjusted advantages and when you apply the mask. The derivation answers my question about Table 3. At reference refresh, the adjusted-advantage update and the pairwise-loss update give proportional auxiliary gradients on filtered responses within each minibatch. The scale factor depends on the minibatch’s valid-token count relative to the average across minibatches, and the gradients match exactly when that count equals the average. I’m keeping my weak-accept recommendation.

We sincerely thank the reviewer for your positive assessment and recognition of our work. We greatly appreciate your constructive feedback throughout the review process, which has helped us further strengthen our manuscript.

Reviewer_GeEKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GPDPO addresses supervision discarded by sign-based conflict filtering in multi-reward policy optimization. A response with conflicting group-relative advantages can still dominate, or be dominated by, another response across reward dimensions. The method retains the GD²PO objective and adds a logistic preference loss on Pareto pairs involving at least one filtered response. It uses mean token log-probability ratios against the sampling policy, detaches retained endpoints in the auxiliary loss, and refreshes the reference after each rollout step. Four API-Bank configurations report higher task accuracy, supported by analyses of recoverable relations, ties, training dynamics, and several design choices.

Critical strengths
  1. The distinction between group-relative conflict and pairwise consistency is useful and clearly demonstrated. In the analyzed two- and three-reward runs, 88.42% and 86.67% of filtered responses retain at least one dominance relation. The method turns this observation into a straightforward auxiliary update without additional response sampling or reward evaluations. Its gradient routing and empty-pair handling are explicitly described. [pp. 3–6, 14–16]

  2. The paper provides positive results in all four tested configurations and meaningful design analyses. Strict dominance, scalarized ordering, a fixed reference, an alternative advantage update, and four values of the auxiliary coefficient are already evaluated. The tie analysis is informative: it explains why requiring strict improvement in every dimension would discard much of the available supervision. These results support the plausibility of recovering useful information from filtered responses. [pp. 7–9, 16]

Critical weaknesses
  1. The principal reservation is the strength of the demonstrated incremental benefit. Relative to the strongest task-accuracy baseline in each configuration, the gains amount to 15, 3, 5, and 7 additional correct answers out of 597. Appendix A.1 states that each main-table entry is a single evaluation of a selected run. This does not establish that the improvements are noise, but it leaves their repeatability and sensitivity to selection unclear. Because the methodological contribution is an incremental way to recover supervision, reliable additional benefit is particularly important to the acceptance case. A new task suite is not required to resolve this concern. [pp. 7, 13]

  2. The fixed-reference comparison does not isolate the effect of refreshing the reference. It simultaneously changes beta from 0.1 to 32 and lambda from 0.1 to 0.0003125. Matching lambda times beta matches the zero-margin derivative, but changes the zero-margin weighted curvature by a factor of 320. Thus the result cannot uniquely support the reference-refresh explanation. This limits interpretation of that design choice, without showing that refreshing is ineffective or invalidating the other ablations. [pp. 8–9, 13]

Questions for authors
  1. How were the reported runs and hyperparameters selected? Can you quantify uncertainty in the key differences against the strongest accuracy baseline, using any independent runs and paired per-question outcomes available? Please distinguish training variability from evaluation uncertainty. [pp. 7, 13]

  2. Can the reference-refresh conclusion be supported with matched beta and lambda, or equally budgeted tuning for both reference choices? Please also specify the exact alternative advantage-adjustment formula, so that its comparison can be interpreted. [pp. 8–9, 13]

3 comments
AuthorsOpenReview ↗

Response to Reviewer GeEK (1/2)

We thank the reviewer for the constructive feedback.

Q1. Run and Hyperparameter Selection, and Uncertainty in Accuracy Gains

Run and hyperparameter selection. To ensure a fair comparison, all methods share the same training and evaluation settings. Shared hyperparameters are held fixed across methods, and method-specific hyperparameters follow each method’s official implementation (Table R1). Each method was trained exactly once per configuration (apart from the ablation and sensitivity runs), and for every method we report the final step-200 checkpoint. No run or checkpoint was selected based on API-Bank results, and we will replace “selected” with “final” in Section 4.1, Appendix A.1, and Tables 4–5. For GPDPO, the pairwise-loss parameters were set to λ = 0.1 and β = 0.1 without benchmark-specific hyperparameter tuning, and the λ sweep in Figure 6 was run afterwards as a sensitivity analysis. Following previous work on tool calling (ToolRL, GDPO, and GD²PO), we use greedy decoding for evaluation. Since greedy decoding is deterministic for a fixed checkpoint, each entry in Tables 1 and 2 is a single score, and we complement these scores with repeated stochastic-decoding evaluation.

Table R1. Training and evaluation settings from Appendix A.1, Tables 4 and 5. Batch sizes are measured in prompts for training.

SettingValue
Training
Training hardwareOne node, 4× NVIDIA RTX 5880 Ada (48 GB each)
Base modelsQwen3-1.7B, Qwen3-4B
Training dataRLLA-4K, 3,920 records before prompt filtering
Max prompt / response length2,048 / 1,024 tokens
Train batch / mini-batch size512 / 128 prompts
Rollout group size GG4 (2,048 responses per rollout step)
Training steps / PPO epochs200 rollout steps / 1 epoch per step
Optimizer / learning rateAdamW / 10−610^{-6}, constant, no warmup
Temperature / top-p / top-k1.0 / 1.0 / disabled
PPO clip range / dual-clip factor(ϵlow,ϵhigh)=(0.2,0.2)(\epsilon_{\mathrm{low}},\epsilon_{\mathrm{high}})=(0.2,0.2) / 3.0
Policy-loss aggregationToken mean
Dynamic batchingEnabled; response groups remain intact
vLLM GPU memory utilization0.60
KL penalty / entropy bonusNeither used
Rewards2R: correctness + length; 3R: additionally format
Pairwise loss parametersλ=0.1\lambda =0.1, β=0.1\beta =0.1; uniform pair weights
Reference policySampling policy; refreshed every rollout step
Checkpoint interval / main resultsEvery 50 steps / final step-200 checkpoint
Evaluation
Evaluation hardwareOne NVIDIA RTX A6000 (48 GB)
DatasetProcessed API-Bank, 597 questions
Level 1 / 2 / 3 questions399 / 67 / 131
DecodingGreedy, one response per question
Temperature / top-p0.0 / 1.0
Max new tokens / total sequence length4,096 / 4,096
Generation batch / tensor parallel size1 / 1
Evaluation seed42
MetricsLevel accuracy, Correct Acc., Length Rew.; Format Acc. for 3R

Training variability and evaluation uncertainty. Consistent with prior work on tool-calling RL (ToolRL, GDPO, and GD²PO), our main results come from a single training run per configuration. Because multi-seed training of all methods was not feasible within the rebuttal period, the analyses below quantify evaluation uncertainty (i.e., variation due to decoding randomness) for these fixed checkpoints and do not capture training variability across different seeds. We will report results over multiple training seeds in the final version.

Repeated evaluation with stochastic decoding. Following your advice, we quantified this evaluation uncertainty by conducting 16 rounds of evaluation using stochastic decoding (temperature = 0.6, top-p = 0.95, top-k disabled). GPDPO consistently achieved the highest mean Correct Acc. in both settings, outperforming GRPO by 3.43 and 2.20 percentage points in the 2R and 3R settings, respectively. Notably, even compared with the strong and recent state-of-the-art baseline GD²PO, GPDPO maintained its performance advantage, achieving improvements of 1.36 percentage points in 2R (64.75% vs. 63.39%) and 0.25 percentage points in 3R (63.55% vs. 63.30%). Furthermore, GPDPO achieved the highest mean Overall score in both settings, indicating that its gains over the strongest accuracy baseline are reliable and stable across repeated evaluations. The results are shown in Tables R2 and R3.

AuthorsOpenReview ↗

Response to Reviewer GeEK (2/2)

Table R2. API-Bank · Qwen3-1.7B (2R)

MethodLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Length Rew.Overall
GRPO67.23 ± 0.9854.76 ± 2.4246.66 ± 1.6061.32 ± 0.780.9665 ± 0.00201.580 ± 0.008
GDPO67.09 ± 0.9558.77 ± 2.1042.56 ± 1.5460.77 ± 0.730.9716 ± 0.00161.579 ± 0.007
GD²PO69.63 ± 0.6761.57 ± 2.0045.32 ± 1.1563.39 ± 0.480.9806 ± 0.00131.614 ± 0.005
GPDPO70.97 ± 0.9462.97 ± 2.2646.71 ± 0.8964.75 ± 0.620.9810 ± 0.00141.628 ± 0.007

Table R3. API-Bank · Qwen3-1.7B (3R)

MethodLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Format Acc. (%)Length Rew.Overall
GRPO67.53 ± 0.7960.45 ± 2.4442.99 ± 1.5761.35 ± 0.7199.82 ± 0.180.9553 ± 0.00222.567 ± 0.010
GDPO66.51 ± 0.8855.50 ± 2.3943.65 ± 1.5160.26 ± 0.7299.96 ± 0.070.9580 ± 0.00252.560 ± 0.007
GD²PO69.38 ± 0.7661.57 ± 2.0845.66 ± 1.7963.30 ± 0.8599.97 ± 0.070.9743 ± 0.00182.607 ± 0.010
GPDPO70.05 ± 0.9062.13 ± 1.9644.47 ± 1.4363.55 ± 0.7599.92 ± 0.110.9780 ± 0.00142.613 ± 0.008

Paired per-question outcome analysis. In the 2R setting, we counted the number of correct responses from GPDPO and GD²PO for each test question across the 16 evaluation rounds. For each question, we then applied a one-sided Fisher’s exact test to these counts to assess whether GPDPO was more likely than GD²PO to answer correctly (nominal significance level of 0.05). As shown in Table R4, GPDPO was significantly more accurate than GD²PO on 19 questions, providing further question-level evidence of consistent gains across repeated evaluations.

Table R4. Per-question comparison with GD²PO (2R)

LevelTotal questionsQuestions favoring GPDPO (p<0.05p < 0.05)
Level 139913
Level 2674
Level 31312
Total59719

Q2. Fixed-Reference Ablation with Matched β and λ, and the Advantage-Adjustment Formula

Thank you for this helpful suggestion.

Fixed-reference ablation with matched β and λ. As suggested, we conducted the fixed-reference ablation with β=λ=0.1\beta =\lambda =0.1. Table R5 shows that, under these matched hyperparameters, full GPDPO still achieves higher task accuracy and a higher Overall score than the fixed-reference variant. This result supports the conclusion in our manuscript that refreshing the reference improves performance. We will add this comparison to Table 3 in the revised version.

Table R5. Fixed-reference ablation on Qwen3-1.7B (2R)

MethodβλLevel 1 Acc. (%)Level 2 Acc. (%)Level 3 Acc. (%)Correct Acc. (%)Format Acc. (%)Length Rew.Overall
Full GPDPO (refreshed reference)0.10.170.9361.1948.8564.9999.330.98121.631
GPDPO (fixed reference, original Table 3)320.000312571.1858.2139.6962.8199.330.97671.605
GPDPO (fixed reference, matched hyperparameters)0.10.169.6761.1941.2262.4898.320.98191.607

Advantage-adjustment formula. In the advantage-adjustment ablation, each valid token tt of a filtered response yny_n receives An,tadj=(λβ/2)(dn/NE)(T^/∣yn∣)A_{n,t}^{\mathrm{adj}}=(\lambda \beta /2)(d_n/N_{E})(\widehat{T}/|y_n|). Here NEN_{E} is the number of Pareto pairs in the rollout batch that contain at least one filtered response, and dnd_n is the number of these pairs that yny_n wins minus the number it loses. We write ∣yn∣|y_n| for the number of valid tokens in yny_n and T^\widehat{T} for the total number of valid response tokens in the rollout batch divided by four, the number of PPO mini-batches. We then incorporate this adjusted advantage into training through the standard PPO loss. In Table 3, Pareto-guided advantage adjustment raises accuracy from 62.48% (GD²PO) to 63.82%, showing that the recovered Pareto relations provide useful supervision under this alternative update. Full GPDPO reaches 64.99%, suggesting that directly optimizing pairwise preference margins makes better use of the recovered supervision.

Dear Reviewer GeEK,

Thank you for your valuable feedback. In our responses, we addressed your questions about how we selected checkpoints and hyperparameters, the uncertainty in the reported accuracy gains, the fixed-reference ablation with matched β and λ, and the formula used for advantage adjustment.

As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.