acceptodds
Under review as a conference paper at ICLR 2027

DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Abstract

Multi-reward reinforcement learning for large language models combines task accuracy with auxiliary objectives such as length and format compliance. In Group Relative Policy Optimization, Reward Combination normalizes the scalarized reward, while Advantage Combination normalizes each reward separately before applying fixed coefficients. The latter removes differences in reward dispersion before combination, even as those statistics change during training. We propose **D**ynamic **V**ariance-adaptive **A**dvantage **O**ptimization (**DVAO**), which adapts the coefficients of normalized advantages using each reward's empirical standard deviation within a rollout group. We prove that DVAO's pointwise advantage magnitude is bounded by that of Reward Combination and characterize the cross-objective coupling in its sensitivity to reward perturbations. An equivalent groupwise rescaling representation relates this magnitude to reward correlations. Experiments on mathematical reasoning and tool use cover four *Qwen3* and *Qwen2.5* model settings. DVAO achieves the highest average accuracy among the compared multi-reward methods in each setting, with length compliance above 99.9% on both math models and the highest average format compliance on both tool-use models.

Primary area:
generative models
Keywords:
Large Language Models, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aRkSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-reward advantage estimation in GRPO. DVAO combines independently normalized advantages using weights proportional to their within-group reward standard deviations. The paper provides advantage-magnitude and reward-sensitivity analyses, followed by experiments on mathematical reasoning and tool use with Qwen models, comparing against RC, AC, and GDPO.

Critical strengths

1.A simple modification with a plausible practical benefit. The method requires little additional computation and no learned auxiliary component. Its normalization can attenuate updates from groups whose reward components partially cancel. This is a useful property to investigate, and the implementation cost makes even a modest, reproducible improvement consequential. 2.Evaluation of joint performance rather than correctness alone. The reported results improve correctness together with auxiliary compliance across both task families. Evaluating the trade-off is appropriate for the problem and provides meaningful evidence of practical usefulness.

Critical weaknesses

1.The theoretical interpretation is stronger than the analysis supports. DVAO can be rewritten as RC multiplied by a group-dependent scaling factor, preserving advantage signs and relative magnitudes within each group. This supports an interpretation of adaptive group weighting, but does not establish improved handling of objective conflicts. Likewise, reward-sensitivity derivatives do not directly establish a policy-gradient regularizer when advantages are detached. The magnitude bounds demonstrate attenuation rather than guaranteed training stability. Equation 4 also appears to give an incorrect gradient for the saturated branch of the clipped objective. These issues weaken the theoretical explanation of the method's benefits. 2.The empirical comparison needs clarification and stronger controls. Table 1 reports 13.41% average mathematical accuracy for GDPO on Qwen3-4B, while its points in Figure 3(a) lie around 36-40%, despite the sweep including equal weights. The evaluation or checkpoint differences behind this discrepancy need clarification. The experiments also do not clearly separate adaptive group weighting from changes in advantage scale. Controlled normalization comparisons, comparable learning-rate tuning, and results across independent training runs would help establish the reliability and source of the reported gains.

Questions for authors

1.Implementation clarification: What is the exact implemented advantage formula, including numerical stabilizers and handling of constant-reward groups? Are rewards, standard deviations, and advantages detached during policy optimization? This determines how closely the group-rescaling identity above describes the actual method. 2.Experimental clarification: Do Table 1 and Figure 3(a) use identical evaluation datasets, aggregation, and checkpoint-selection rules? Please reconcile the GDPO results and provide the numerical sweep results with their corresponding weights. 3.Evidence that would affect my recommendation: Does DVAO retain its advantage over RC when overall advantage scale is controlled and learning rates receive comparable tuning? Results across independent training seeds would help establish whether the improvement is reproducible. A constant-scale control and shuffled group-scaling factors would be particularly informative for separating adaptive group weighting from general attenuation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aRkS (Part 1/2)

Thank you for your careful review. We clarify the theoretical questions and report the requested experimental checks below.


W1 / Q1: Implementation details and theoretical interpretation.

For query xix_i and its GG sampled responses, we compute

\sigma_k^i=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_k^{(i,j)}-\mu_k^i)^2},$$ $$A_{\mathrm{DVAO}}^{(i,j)} =\frac{\sum_k w_k(r_k^{(i,j)}-\mu_k^i)} {\sum_k w_k\sigma_k^i+\epsilon}.$$ We add $\epsilon=10^{-6}$ only to the final denominator and form the centered numerator directly. Constant channels contribute zero to the numerator and unstabilized denominator. If all positively weighted channels are constant, all group advantages are zero. Rewards, means, standard deviations, and advantages are detached. No additional batch-level normalization or whitening is applied. DVAO adaptively rescales RC across groups while preserving within-group advantage signs and proportions (Section 4.2 and Appendix E, under the analytical normalization definitions). The controls below test the value of this allocation. Proposition 3 characterizes **reward sensitivity**. These derivatives do not enter the policy gradient as an additional regularization term because advantages are held fixed during optimization (Section 4.2). Proposition 2 bounds advantage magnitudes without establishing training stability or resolution of policy-gradient conflicts. Equation 4 differentiates the **unclipped surrogate** $J_{\text{unc}}$ defined immediately before it, with sampled responses and advantages held fixed. For the clipped objective, a locally constant saturated branch contributes zero gradient. --- **W2 / Q2: GDPO results in Table 1 and Figure 3(a).** Table 1 and Figure 3(a) use identical evaluation datasets, aggregation, and checkpoint-selection rules. **GDPO's equal-weight result is 13.41% accuracy and 97.81% length compliance**, matching Table 1. Figure 3(a) restricts accuracy to [36, 46]: the visible GDPO point has accuracy weight 0.1 and scores 39.17%/94.29%, while the equal-weight point and the other three points lie outside this range. The full mathematical-reasoning sweep is below. Each cell reports accuracy / length compliance in percent, and the length weight is $1-w_{\mathrm{acc}}$. | Accuracy weight | RC | AC | GDPO | DVAO | |---|---|---|---|---| | 0.1 | 39.01 / 95.48 | 38.60 / 86.76 | 39.17 / 94.29 | 42.43 / 99.81 | | 0.3 | 39.41 / 95.13 | 39.14 / 95.11 | 18.07 / 97.33 | 41.35 / 99.31 | | 0.5 | 38.99 / 96.39 | 38.75 / 96.23 | 13.41 / 97.81 | 42.19 / 99.91 | | 0.7 | 39.26 / 97.08 | 38.90 / 97.21 | 8.77 / 99.66 | 45.15 / 99.99 | | 0.9 | 39.37 / 97.62 | 9.50 / 99.38 | 5.42 / 99.10 | 39.53 / 99.85 | GDPO uses the official implementation integrated into verl.
AuthorsOpenReview ↗

Response to Reviewer aRkS (Part 2/2)

W2 / Q3: Scale controls, learning rates, and independent training runs.

DVAO remains ahead of both RMS-matched controls. It also outperforms RC at all three tested learning rates, and the five-run comparison supports reproducible gains.

We added two scale controls on Qwen3-4B-Base under Table 1's training and evaluation configuration. For the batch-uniform control, we compute RC and DVAO advantages on the control's own full rollout batch and set

A_{\mathrm{control}}^{(i,j)}=c_t A_{\mathrm{sum}}^{(i,j)}.$$ The coefficient is recomputed per batch and shared across groups, matching DVAO's response-level advantage RMS while preserving RC's relative group scales. An all-zero RC batch retains zero advantages. The shuffled control randomly permutes scaling factors among groups with nonzero RC signal, then applies a common rescaling to match DVAO's advantage RMS on that control's own samples. This breaks the group-factor correspondence while preserving the factor distribution before RMS matching. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC, Table 1 | 38.99 | 96.39 | | DVAO, Table 1 | 42.19 | 99.91 | | Batch-uniform, RMS-matched RC | 39.35 | 97.14 | | Shuffled group scaling, RMS-matched | 38.59 | 96.13 | DVAO improves both metrics over both RMS-matched controls, supporting allocation according to each group's own reward statistics beyond overall attenuation or arbitrary group scaling. We compared RC and DVAO on the same three learning rates below, keeping Qwen3-4B-Base, all other training settings, and five-benchmark mean@16 evaluation fixed. Each entry is a single training run. The $10^{-6}$ entries reuse Table 1. Cells report accuracy / length compliance in percent. | Learning rate | RC | DVAO | |---|---:|---:| | $5\times10^{-7}$ | 38.43 / 95.86 | 42.46 / 99.72 | | $10^{-6}$ | 38.99 / 96.39 | 42.19 / 99.91 | | $2\times10^{-6}$ | 38.65 / 95.71 | 42.24 / 99.37 | DVAO achieves higher accuracy and length compliance at all three tested learning rates. We added four independent training runs per method to the original Table 1 run, giving five each for RC and DVAO at the fixed Qwen3-4B-Base main configuration. Results report mean ± sample standard deviation. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC | 39.00 ± 0.32 | 95.59 ± 0.71 | | DVAO | 42.27 ± 0.37 | 99.78 ± 0.12 | DVAO improves mean accuracy by 3.27 percentage points and mean length compliance by 4.19 points, calculated from unrounded means, supporting reproducible joint gains in this setting. Thank you again for your careful review and constructive suggestions. We would be grateful if you could revisit your assessment in light of these results and the theoretical clarifications.
AuthorsOpenReview ↗

Follow-up on our response and additional experiments

Dear Reviewer aRkS,

As the discussion period draws to a close, we would like to follow up on our response and ask whether any concerns remain that we could further clarify.

Following your suggestions, we provided RMS-matched uniform and shuffled scaling controls, a matched learning-rate comparison, and results from five independent training runs per method. These results support DVAO’s gains beyond overall advantage scaling and across the tested learning rates and training runs. We also clarified the theoretical interpretation and implementation details, and reconciled the GDPO table–figure discrepancy.

We would welcome any further questions and would be happy to discuss them during the remaining time. If these clarifications and additional results address your concerns, we would be grateful if you could revisit your assessment in light of this evidence.

Thank you again for your time and constructive feedback.

AuthorsOpenReview ↗

Final follow-up before the discussion closes

Dear Reviewer aRkS,

With only a few hours left in the discussion, we wanted to follow up once more. Thank you for the specific questions and experiments you suggested.

Your observation that DVAO rescales RC within each group helped us clarify the theory. We provided the exact normalization formula and explained what the reward-sensitivity analysis establishes when advantages are detached. We also reconciled the GDPO results. The equal-weight accuracy of 13.41% matches Table 1 and falls below the range plotted in Figure 3(a). The visible point at 39.17% uses an accuracy weight of 0.1.

In the experiments you suggested, DVAO outperformed both RMS-matched controls on accuracy and length compliance. It also outperformed RC on both metrics at all three learning rates tested. At the main learning rate, five independent runs per method gave mean gains over RC of 3.27 percentage points in accuracy and 4.19 points in length compliance.

If any concern remains, we would welcome a brief note while there is still time to respond. If these clarifications and results address your concerns, we would be grateful if you would reconsider your recommendation.

Reviewer_ALjrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates multi-reward reinforcement learning for LLM alignment, focusing on how multiple reward signals are combined in GRPO-based training. The authors discuss two existing approaches: Reward Combination (RC), which combines rewards before normalization, and Advantage Combination (AC), which normalizes rewards separately before combining their advantages with fixed coefficients. To address their limitations, the paper proposes Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts the coefficients of normalized advantages based on each reward's empirical standard deviation within a rollout group. The paper presents theoretical comparisons of advantage magnitudes and reward sensitivities, followed by experiments on mathematical reasoning and tool-use tasks across multiple model settings.

Critical strengths
  1. The paper addresses multi-reward optimization in RL-based LLM alignment, an important topic for balancing task accuracy with auxiliary objectives such as length and format compliance.
  2. The paper provides useful theoretical characterizations of RC and AC, including their differences in advantage magnitudes and gradient decomposition. Propositions 1–3 further clarify how DVAO differs from these methods in advantage scaling and reward sensitivity.
  3. The experiments cover multiple model settings and two task categories, mathematical reasoning and tool use. The reported results in Tables 1 and 2 demonstrate promising improvements over existing multi-reward optimization methods.
Critical weaknesses

The paper considers two existing approaches, RC and AC, but the motivation presented in the abstract and introduction primarily emphasizes the limitations of AC, particularly its use of fixed coefficients. Although Propositions 1–3 provide theoretical comparisons among these approaches, the practical limitations of RC are not clearly explained. Proposition 1 shows that RC produces larger or equal mean squared advantages than AC, but the implications of this result for optimization remain unclear.

Questions for authors

Could the authors clarify the specific limitations of RC that motivate DVAO? In particular, is Proposition 1 intended to provide theoretical evidence of RC's limitations? Since RC produces larger mean squared advantages than AC, does this lead to larger policy-gradient updates or potentially unstable training? Further explanation or empirical evidence would help clarify this connection.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ALjr

Thank you for your feedback. We clarify RC's limitation, Proposition 1's implications, and the supporting evidence below.


W1 / Q1: RC motivation and the implications of Proposition 1.

Exploding gradients and abrupt gradient-norm growth can disrupt optimization [1, 2]. Proposition 1 compares advantage scales. Policy-gradient norms also depend on score vectors and their cancellation, so this comparison alone does not imply larger policy-gradient norms or training instability.

RC normalizes each nondegenerate group's combined reward to unit mean squared advantage, even when its reward components partially cancel. DVAO retains this information in the relative group scales. Under the analytical definitions, for Si=∑kwkσkiS_i=\sum_k w_k\sigma_k^i and σsumi>0\sigma_{\mathrm{sum}}^i>0,

=\lambda_i A_{\mathrm{sum}}^{(i,j)},\qquad \lambda_i=\frac{\sigma_{\mathrm{sum}}^i}{S_i}\in(0,1].$$ Here, $A_{\mathrm{sum}}^{(i,j)}$ is the RC advantage. DVAO preserves within-group advantage signs and proportions while adapting relative group contributions. In the fixed-rollout, unclipped analysis with advantages held fixed, the aggregate gradient changes from $\sum_i g_{\mathrm{RC}}^i$ to $\sum_i\lambda_i g_{\mathrm{RC}}^i$, where $g_{\mathrm{RC}}^i$ is the group's RC gradient contribution. Thus, group-dependent scaling can change the aggregate update direction. We tested this allocation with two controls on Qwen3-4B-Base under Table 1's training and evaluation configuration: uniform RC scaling and randomly shuffled DVAO scaling factors among groups with nonzero RC signal. In each full rollout batch, both controls use a common rescaling to match DVAO's response-level advantage RMS computed on the control's own samples. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC, Table 1 | 38.99 | 96.39 | | Batch-uniform, RMS-matched RC | 39.35 | 97.14 | | Shuffled group scaling, RMS-matched | 38.59 | 96.13 | | DVAO, Table 1 | 42.19 | 99.91 | DVAO improves both metrics over both RMS-matched controls, supporting the value of assigning group scales according to each group's own reward statistics. Thank you again for your thoughtful feedback. We hope this response clarifies the motivation for DVAO and the role of Proposition 1. [1] Pascanu, R., Mikolov, T., and Bengio, Y. *On the difficulty of training recurrent neural networks*. ICML 2013. [2] Takase, S., et al. *Spike No More: Stabilizing the Pre-training of Large Language Models*. COLM 2025.
Reviewer_nS68OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on multi-reward RL for LLMs, where task accuracy is optimized together with auxiliary requirements such as length or format compliance. The authors observe that AC and GDPO normalize each reward separately and then combine them with fixed weights, which removes the differences in how much each reward varies within a group. To address this problem, the authors propose DVAO, which adjusts the weight of each normalized reward by its standard deviation within the group. As a result, a reward that no longer varies within a group, such as a length requirement that all responses already satisfy, automatically receives a smaller weight, and no additional hyperparameter is introduced. Experiments on mathematical reasoning and tool use show that DVAO outperforms the other multi-reward baselines in average accuracy while keeping high compliance with the auxiliary requirements.

Critical strengths
  1. In this paper, the authors design the experiments reasonably and provide comprehensive comparisons. The authors evaluate the method on two types of tasks, several model families and sizes, and compare it with both single-reward and multi-reward baselines. The authors also study different reward weights, group sizes and a setting with three rewards. In addition, the authors provide sufficient details on the data, hyperparameters, hardware and evaluation protocol, so the reproducibility of the paper is good.

  2. The authors present the method with good visualization and concrete examples. The training dynamics are shown in figures, and the authors use simple examples, such as a length reward whose weight drops when all responses in a group satisfy the requirement and recovers when compliant and non-compliant responses coexist, to illustrate how the method behaves. This makes the paper readable and easy to catch up on.

Critical weaknesses
  1. The authors do not discuss in enough detail the most direct characterization of the method. In Section 4.2, the authors state in one sentence that DVAO is equivalent to multiplying the advantages of the sum-first scheme by a factor between zero and one within each group, so the direction of the advantages within a group is the same as in the sum-first scheme and only the relative scale across groups changes. However, the derivation and its relation to reward correlations are given only in the appendix, and the method is still mainly described as reintroducing dispersion differences into the normalize-first scheme. The authors may consider expanding this equivalence in the main text, since it is the most direct way for readers to understand what the method changes.

  2. The authors do not explain why changing the relative scale across groups leads to clear improvements over the sum-first scheme. According to the authors' own statistics, the groups in which the scaling factor is clearly below one form a limited and decreasing fraction of the data during training, yet the gains over the sum-first scheme are noticeable in several settings. The authors may consider providing a mechanism-level explanation or an analysis that connects the group-wise scaling to the observed improvements.

Questions for authors
  1. The scaling factor is smaller in groups where the rewards are less correlated, for example when correctness and length compliance are negatively correlated within a group. This means that DVAO reduces the weight of groups that contain conflicting signals. Do the authors consider this to be the main source of the improvement, and how does this relate to the motivation of reintroducing dispersion differences?

  2. Most rewards in the experiments are binary, and only one setting uses three rewards, including one continuous reward. How would the distribution of the scaling factor change, and would the method remain effective, when more of the rewards are continuous and the number of rewards is larger?

  3. In the training dynamics, the response length under DVAO becomes longer and oscillates in the later stage. Once the length reward is saturated and its weight approaches zero, will the response length continue to grow until it reaches the limit, and could this become a risk in longer training?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nS68 (Part 1/2)

Thank you for your thoughtful review of the mechanism and experimental evidence.


W1 / W2 / Q1: Group rescaling, reward correlations, and the source of gains.

The added controls support scaling each group according to its own reward statistics as one contributor to DVAO's gains.

DVAO's dispersion-based coefficients produce adaptive group scaling relative to RC. For a group with Si=∑kwkσki>0S_i=\sum_k w_k\sigma_k^i>0, DVAO sets each channel's coefficient to w~ki=wkσki/Si\widetilde w_k^i=w_k\sigma_k^i/S_i. Multiplying this coefficient by the normalized reward (rk(i,j)−μki)/σki(r_k^{(i,j)}-\mu_k^i)/\sigma_k^i restores the dispersion of active channels. Constant channels contribute zero. Under the analytical definitions, for nondegenerate RC groups,

=\frac{\sum_k w_k(r_k^{(i,j)}-\mu_k^i)}{S_i} =\frac{\sigma_{\mathrm{sum}}^i}{S_i}A_{\mathrm{sum}}^{(i,j)}.$$ Here, $A_{\mathrm{sum}}^{(i,j)}$ denotes the RC advantage, and $\lambda_i=\sigma_{\mathrm{sum}}^i/S_i$ is the group scale. DVAO preserves within-group advantage signs and proportions while adapting relative scales across groups. We will expand Section 4.2 with the rescaling and correlation derivations from Appendix E. Let $\hat{\rho}_{kl}^i$ denote the within-group reward correlation. With sums over active channels, $$(\lambda_i)^2 =\sum_k (\widetilde w_k^i)^2 +2\sum_{k<l}\widetilde w_k^i\widetilde w_l^i\hat{\rho}_{kl}^i.$$ Holding the component dispersions and weights fixed, lower reward correlation decreases the group scale, including when correlations remain positive. In Section 5.5, the jointly active Qwen3-8B validation groups have mean correlation 0.267 and mean $\lambda_i=0.801$, with negative correlation in 6.82% of groups. The jointly active fraction falls from 40.52% at step 0 to 9.56% at step 500, so the modification is more prevalent early in training. Groups with one active reward have scale one under the analytical definitions, while all-constant groups carry no centered reward signal. Group frequency alone does not determine the optimization effect. In the fixed-rollout, unclipped analysis, DVAO scales each nondegenerate RC group's gradient contribution by $\lambda_i$, so the aggregate update also depends on the affected contributions' magnitudes and directions. Earlier updates can alter later on-policy samples. We evaluated two controls on Qwen3-4B-Base under Table 1's training and evaluation configuration. For each full rollout batch, both match the response-level RMS of DVAO advantages computed on the control's own samples. The batch-uniform control recomputes a single coefficient, preserving RC's relative group scales. The shuffled control randomly permutes factors among groups with nonzero RC signal, then applies a common rescaling. The permutation preserves the factor distribution before RMS matching while breaking the group–factor correspondence. The table compares these controls with the original Table 1 results. | Method | Average accuracy (%) | Average length compliance (%) | |---|---:|---:| | RC, Table 1 | 38.99 | 96.39 | | DVAO, Table 1 | 42.19 | 99.91 | | Batch-uniform, RMS-matched RC | 39.35 | 97.14 | | Shuffled group scaling, RMS-matched | 38.59 | 96.13 | DVAO achieves higher accuracy and length compliance than both RMS-matched controls. --- **Q2: More continuous rewards and more objectives.** We are working hard to complete additional experiments comparing RC and DVAO with multiple continuous reward channels and more objectives. These experiments are not yet complete because they require substantial additional training and evaluation time. We will report their results and analyses at a later stage. In the meantime, we address the question using the general scaling expression and our existing three-reward results. The correlation expression above applies to continuous as well as binary rewards. The scale distribution depends jointly on component dispersions, base weights, and correlations. For example, under the analytical definitions, $K$ active channels with equal $w_k\sigma_k^i$ and zero pairwise correlation give $\lambda_i=1/\sqrt K$, whereas perfectly positively correlated active channels give $\lambda_i=1$. Reward units and calibration also matter because the coefficients explicitly depend on reward standard deviations. Section 5.6 and Table 6 evaluate three rewards, including continuous correctness-gated brevity. DVAO obtains 41.70% accuracy, 97.27% length compliance, and a brevity score of 34.15, exceeding RC by 4.97 and 2.51 percentage points on the first two metrics. These results support DVAO's effectiveness in the evaluated mixed binary–continuous reward setting. For the additional experiments, we will report per-objective performance and scaling-factor distributions, together with the reward definitions, scales, and base weights.
AuthorsOpenReview ↗

Response to Reviewer nS68 (Part 2/2)

Q3: Late-stage length growth and longer training.

The binary length reward assigns the same value to every response at or below 4,000 tokens, so length growth within that range leaves the reward unchanged. DVAO recomputes the length coefficient from each new rollout group. All-compliant and all-violating groups both have constant length rewards and provide no length-specific relative signal. The length channel receives a zero coefficient in those groups and becomes active again when compliant and noncompliant responses coexist.

After 500 steps, final mean@16 length compliance is 99.91% on Qwen3-4B and 99.92% on Qwen3-8B. The reported dynamics show later-stage length growth alongside high compliance over this horizon. Whether this growth persists beyond 500 steps requires longer-horizon measurements.

The longer-training comparison will extend RC and DVAO beyond 500 steps, tracking response length, accuracy, length compliance, and the length-channel coefficient. We will report these trajectories and the final results. We are also working hard on the longer-training extension, which requires additional training and evaluation time, and will report its results and analyses at a later stage.

Thank you again for your constructive feedback. We appreciate your time and welcome any further questions or suggestions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your careful reviews. Your suggestions helped us test why DVAO improves both accuracy and length compliance and how reliably those gains hold.

Does group-specific scaling account for some of the gains? On Qwen3-4B-Base, we added two controls that match DVAO’s response-level advantage RMS on each control’s own rollout batch. The first applies one scale to all RC groups in a batch. The second shuffles scaling factors among groups with nonzero RC signal before matching RMS.

MethodAccuracy (%)Length compliance (%)
Reward Combination (RC), Table 138.9996.39
RC with uniform scaling, new control39.3597.14
Shuffled group scaling, new control38.5996.13
DVAO, Table 142.1999.91

Uniform scaling improves both metrics over RC. DVAO gains a further 2.84 percentage points in accuracy and 2.77 in length compliance over that control. Its advantage over shuffled scaling also supports assigning each group a scale based on its own reward statistics.

Do the gains hold across training runs and learning rates? At the main learning rate, we added four independent runs per method to the original Table 1 run. The table reports the mean and sample standard deviation across all five runs per method on Qwen3-4B-Base.

MethodAccuracy (%)Length compliance (%)
RC39.00 ± 0.3295.59 ± 0.71
DVAO42.27 ± 0.3799.78 ± 0.12

DVAO’s mean gains over RC were 3.27 percentage points in accuracy and 4.19 in length compliance. In a separate matched learning-rate sweep, DVAO scored higher on both metrics at all three rates tested (5 × 10⁻⁷, 10⁻⁶, and 2 × 10⁻⁶).

Theory, implementation, and evaluation. We clarified that Propositions 1 and 2 concern advantage magnitude. Policy-gradient norms also depend on the directions of the gradient contributions. Proposition 3 describes sensitivity to the observed rewards, and Equation 4 differentiates the unclipped surrogate with fixed advantages. We also provided the exact implemented normalization and the full GDPO weight sweep. The equal-weight GDPO result agrees with Table 1. Its 13.41% accuracy falls outside the range shown in Figure 3(a).

We appreciate the time you have devoted to the discussion and your consideration of these results.

AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair and Reviewers,

DVAO uses the variation in each reward across answers to the same prompt to decide how much that reward contributes to training. A length reward, for example, gets zero weight in a group where every answer meets the limit. The rule adds no learned component or tuning parameter beyond the base reward weights.

In the submitted paper, DVAO achieved the highest average accuracy among the compared multi-reward methods in all four Qwen settings. Length compliance exceeded 99.9% on both math models. On BFCL-v4 with Qwen2.5-3B-Instruct, DVAO improved on Reward Combination (RC) in both accuracy (51.02% to 56.66%) and format compliance (60.48% to 76.65%). The main tables reported one training run per method. (Table 1 and 2)

Reviewer aRkS

Question raisedOur response and evidence
Q1. What is implemented, and what does the theory establish?We gave the exact normalization formula, including the numerical stabilizer and handling of constant rewards. Advantages are held fixed during optimization. Proposition 3 measures sensitivity to the observed rewards, and Eq. 4 differentiates the unclipped objective. The gradient is zero on a locally constant clipped branch.
Q2. Why do the GDPO results in Table 1 and Figure 3(a) appear inconsistent?We provided all five reward-weight settings and confirmed the same evaluation and checkpoint rules. The equal-weight accuracy of 13.41% falls below the plotted range. The visible 39.17% point uses an accuracy weight of 0.1.
Q3. Do the gains survive scale controls and comparable learning rates, and do they repeat?On Qwen3-4B-Base math, DVAO beat both controls that matched the root-mean-square advantage (uniform and shuffled scaling). It also beat RC on both metrics at all three tested learning rates. At the main rate, we added four runs per method. Across five runs each, DVAO gained 3.27 percentage points in mean accuracy and 4.19 in mean length compliance over RC.

Reviewer nS68

Question raisedOur response and evidence
Q1. Where do the gains come from? Are negatively correlated groups the main source?With overall advantage scale matched, DVAO beat uniform scaling by 2.84 points in accuracy and 2.77 in length compliance on Qwen3-4B-Base math. Shuffling the factors among groups also lowered both metrics. This supports assigning each group a scale based on its own reward statistics.
Q2. What happens with more objectives and continuous rewards?We showed how the group scale depends on reward dispersion, weights and correlations. The submitted three-reward experiment included one continuous brevity reward and favored DVAO on all three metrics.
Q3. Will response length keep growing with longer training?All answers up to 4,000 tokens receive the same length reward. The length signal returns when compliant and violating answers occur in the same group. At step 500, length compliance was 99.91% on 4B and 99.92% on 8B. Measurements beyond 500 steps are ongoing.

Reviewer ALjr

Question raisedOur response and evidence
What practical limitation of RC motivates DVAO?RC normalizes every nonconstant combined reward to the same within-group scale, even when reward components partly cancel. DVAO retains within-group advantage proportions and can give those groups less weight. It outperformed both scale-matched controls on accuracy and length compliance.
Does Proposition 1 imply larger policy gradients or unstable training?Proposition 1 compares mean squared advantages. The policy-gradient norm also depends on the directions of the individual gradient contributions and how much they cancel. Advantage magnitude alone therefore cannot establish training instability.

We have received no public follow-up from any of the three reviewers and would have welcomed the opportunity to discuss these results with them further. We thank the reviewers for suggesting these tests and would appreciate your consideration of the additional evidence in the meta-review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.