- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-reward advantage estimation in GRPO. DVAO combines independently normalized advantages using weights proportional to their within-group reward standard deviations. The paper provides advantage-magnitude and reward-sensitivity analyses, followed by experiments on mathematical reasoning and tool use with Qwen models, comparing against RC, AC, and GDPO.
Critical strengths
1.A simple modification with a plausible practical benefit. The method requires little additional computation and no learned auxiliary component. Its normalization can attenuate updates from groups whose reward components partially cancel. This is a useful property to investigate, and the implementation cost makes even a modest, reproducible improvement consequential. 2.Evaluation of joint performance rather than correctness alone. The reported results improve correctness together with auxiliary compliance across both task families. Evaluating the trade-off is appropriate for the problem and provides meaningful evidence of practical usefulness.
Critical weaknesses
1.The theoretical interpretation is stronger than the analysis supports. DVAO can be rewritten as RC multiplied by a group-dependent scaling factor, preserving advantage signs and relative magnitudes within each group. This supports an interpretation of adaptive group weighting, but does not establish improved handling of objective conflicts. Likewise, reward-sensitivity derivatives do not directly establish a policy-gradient regularizer when advantages are detached. The magnitude bounds demonstrate attenuation rather than guaranteed training stability. Equation 4 also appears to give an incorrect gradient for the saturated branch of the clipped objective. These issues weaken the theoretical explanation of the method's benefits. 2.The empirical comparison needs clarification and stronger controls. Table 1 reports 13.41% average mathematical accuracy for GDPO on Qwen3-4B, while its points in Figure 3(a) lie around 36-40%, despite the sweep including equal weights. The evaluation or checkpoint differences behind this discrepancy need clarification. The experiments also do not clearly separate adaptive group weighting from changes in advantage scale. Controlled normalization comparisons, comparable learning-rate tuning, and results across independent training runs would help establish the reliability and source of the reported gains.
Questions for authors
1.Implementation clarification: What is the exact implemented advantage formula, including numerical stabilizers and handling of constant-reward groups? Are rewards, standard deviations, and advantages detached during policy optimization? This determines how closely the group-rescaling identity above describes the actual method. 2.Experimental clarification: Do Table 1 and Figure 3(a) use identical evaluation datasets, aggregation, and checkpoint-selection rules? Please reconcile the GDPO results and provide the numerical sweep results with their corresponding weights. 3.Evidence that would affect my recommendation: Does DVAO retain its advantage over RC when overall advantage scale is controlled and learning rates receive comparable tuning? Results across independent training seeds would help establish whether the improvement is reproducible. A constant-scale control and shuffled group-scaling factors would be particularly informative for separating adaptive group weighting from general attenuation.
4 comments
Response to Reviewer aRkS (Part 2/2)
W2 / Q3: Scale controls, learning rates, and independent training runs.
DVAO remains ahead of both RMS-matched controls. It also outperforms RC at all three tested learning rates, and the five-run comparison supports reproducible gains.
We added two scale controls on Qwen3-4B-Base under Table 1's training and evaluation configuration. For the batch-uniform control, we compute RC and DVAO advantages on the control's own full rollout batch and set
A_{\mathrm{control}}^{(i,j)}=c_t A_{\mathrm{sum}}^{(i,j)}.$$ The coefficient is recomputed per batch and shared across groups, matching DVAO's response-level advantage RMS while preserving RC's relative group scales. An all-zero RC batch retains zero advantages. The shuffled control randomly permutes scaling factors among groups with nonzero RC signal, then applies a common rescaling to match DVAO's advantage RMS on that control's own samples. This breaks the group-factor correspondence while preserving the factor distribution before RMS matching. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC, Table 1 | 38.99 | 96.39 | | DVAO, Table 1 | 42.19 | 99.91 | | Batch-uniform, RMS-matched RC | 39.35 | 97.14 | | Shuffled group scaling, RMS-matched | 38.59 | 96.13 | DVAO improves both metrics over both RMS-matched controls, supporting allocation according to each group's own reward statistics beyond overall attenuation or arbitrary group scaling. We compared RC and DVAO on the same three learning rates below, keeping Qwen3-4B-Base, all other training settings, and five-benchmark mean@16 evaluation fixed. Each entry is a single training run. The $10^{-6}$ entries reuse Table 1. Cells report accuracy / length compliance in percent. | Learning rate | RC | DVAO | |---|---:|---:| | $5\times10^{-7}$ | 38.43 / 95.86 | 42.46 / 99.72 | | $10^{-6}$ | 38.99 / 96.39 | 42.19 / 99.91 | | $2\times10^{-6}$ | 38.65 / 95.71 | 42.24 / 99.37 | DVAO achieves higher accuracy and length compliance at all three tested learning rates. We added four independent training runs per method to the original Table 1 run, giving five each for RC and DVAO at the fixed Qwen3-4B-Base main configuration. Results report mean ± sample standard deviation. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC | 39.00 ± 0.32 | 95.59 ± 0.71 | | DVAO | 42.27 ± 0.37 | 99.78 ± 0.12 | DVAO improves mean accuracy by 3.27 percentage points and mean length compliance by 4.19 points, calculated from unrounded means, supporting reproducible joint gains in this setting. Thank you again for your careful review and constructive suggestions. We would be grateful if you could revisit your assessment in light of these results and the theoretical clarifications.Follow-up on our response and additional experiments
Dear Reviewer aRkS,
As the discussion period draws to a close, we would like to follow up on our response and ask whether any concerns remain that we could further clarify.
Following your suggestions, we provided RMS-matched uniform and shuffled scaling controls, a matched learning-rate comparison, and results from five independent training runs per method. These results support DVAO’s gains beyond overall advantage scaling and across the tested learning rates and training runs. We also clarified the theoretical interpretation and implementation details, and reconciled the GDPO table–figure discrepancy.
We would welcome any further questions and would be happy to discuss them during the remaining time. If these clarifications and additional results address your concerns, we would be grateful if you could revisit your assessment in light of this evidence.
Thank you again for your time and constructive feedback.
Final follow-up before the discussion closes
Dear Reviewer aRkS,
With only a few hours left in the discussion, we wanted to follow up once more. Thank you for the specific questions and experiments you suggested.
Your observation that DVAO rescales RC within each group helped us clarify the theory. We provided the exact normalization formula and explained what the reward-sensitivity analysis establishes when advantages are detached. We also reconciled the GDPO results. The equal-weight accuracy of 13.41% matches Table 1 and falls below the range plotted in Figure 3(a). The visible point at 39.17% uses an accuracy weight of 0.1.
In the experiments you suggested, DVAO outperformed both RMS-matched controls on accuracy and length compliance. It also outperformed RC on both metrics at all three learning rates tested. At the main learning rate, five independent runs per method gave mean gains over RC of 3.27 percentage points in accuracy and 4.19 points in length compliance.
If any concern remains, we would welcome a brief note while there is still time to respond. If these clarifications and results address your concerns, we would be grateful if you would reconsider your recommendation.
Response to Reviewer aRkS (Part 1/2)
Thank you for your careful review. We clarify the theoretical questions and report the requested experimental checks below.
W1 / Q1: Implementation details and theoretical interpretation.
For query xi and its G sampled responses, we compute
\sigma_k^i=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_k^{(i,j)}-\mu_k^i)^2},$$ $$A_{\mathrm{DVAO}}^{(i,j)} =\frac{\sum_k w_k(r_k^{(i,j)}-\mu_k^i)} {\sum_k w_k\sigma_k^i+\epsilon}.$$ We add $\epsilon=10^{-6}$ only to the final denominator and form the centered numerator directly. Constant channels contribute zero to the numerator and unstabilized denominator. If all positively weighted channels are constant, all group advantages are zero. Rewards, means, standard deviations, and advantages are detached. No additional batch-level normalization or whitening is applied. DVAO adaptively rescales RC across groups while preserving within-group advantage signs and proportions (Section 4.2 and Appendix E, under the analytical normalization definitions). The controls below test the value of this allocation. Proposition 3 characterizes **reward sensitivity**. These derivatives do not enter the policy gradient as an additional regularization term because advantages are held fixed during optimization (Section 4.2). Proposition 2 bounds advantage magnitudes without establishing training stability or resolution of policy-gradient conflicts. Equation 4 differentiates the **unclipped surrogate** $J_{\text{unc}}$ defined immediately before it, with sampled responses and advantages held fixed. For the clipped objective, a locally constant saturated branch contributes zero gradient. --- **W2 / Q2: GDPO results in Table 1 and Figure 3(a).** Table 1 and Figure 3(a) use identical evaluation datasets, aggregation, and checkpoint-selection rules. **GDPO's equal-weight result is 13.41% accuracy and 97.81% length compliance**, matching Table 1. Figure 3(a) restricts accuracy to [36, 46]: the visible GDPO point has accuracy weight 0.1 and scores 39.17%/94.29%, while the equal-weight point and the other three points lie outside this range. The full mathematical-reasoning sweep is below. Each cell reports accuracy / length compliance in percent, and the length weight is $1-w_{\mathrm{acc}}$. | Accuracy weight | RC | AC | GDPO | DVAO | |---|---|---|---|---| | 0.1 | 39.01 / 95.48 | 38.60 / 86.76 | 39.17 / 94.29 | 42.43 / 99.81 | | 0.3 | 39.41 / 95.13 | 39.14 / 95.11 | 18.07 / 97.33 | 41.35 / 99.31 | | 0.5 | 38.99 / 96.39 | 38.75 / 96.23 | 13.41 / 97.81 | 42.19 / 99.91 | | 0.7 | 39.26 / 97.08 | 38.90 / 97.21 | 8.77 / 99.66 | 45.15 / 99.99 | | 0.9 | 39.37 / 97.62 | 9.50 / 99.38 | 5.42 / 99.10 | 39.53 / 99.85 | GDPO uses the official implementation integrated into verl.