acceptodds
Under review as a conference paper at ICLR 2027

Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

Abstract

Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yYZLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focus on multi-reward RL for LLMs, and in particular on the uneven learning progress across objectives that remains after GDPO normalizes each reward separately. To address this problem, the authors first show that, under idealized normalization, the total advantage signal of a reward over a batch is proportional to the fraction of groups in which this reward actually varies, so rewards that vary in fewer groups contribute less to training. Based on this relation, the authors propose DARA, which computes a weight for each reward from this fraction on every batch and gives larger weights to rewards that vary in fewer groups. Experiments on tool calling and mathematical reasoning show that DARA reaches the target behaviors in fewer training steps than the baselines, while remaining competitive in final performance.

Critical strengths
  1. In this paper, the authors start from the uneven learning reported in GDPO and ask what causes it and whether reward weights can be derived from this cause, so the motivation is clear. The authors also give clear definitions of the key quantities, and the main relation can be derived in a few lines, which makes the core idea of the paper easy to follow.

  2. The authors provide sufficient details for reproduction, including the training and evaluation configurations, the reward definitions, the pseudocode, and the definitions of the key metrics, such as the over-length rate and the joint success rate that requires both correctness and length compliance.

  3. The authors present the method and the results with good visualizations. Figure 1 uses a small example to show how the density, the weights and the energy ratio are computed, together with the training curves of the format reward and its density. The figures on the group size, on the training step at which each compliance level is reached, and on the energy ratio during training further connect the analysis with the experimental results. This makes the paper easy to catch up on.

Critical weaknesses
  1. The results in Table 4a are not consistent with those in Table 3. Both tables report DARA-Asym and DARA-Sym on DeepSeek-R1-1.5B, but the accuracy and the over-length rate in Table 4a are clearly different from those in Table 3, and neither the caption nor the text states which checkpoint Table 4a is taken from. The authors may consider stating the training step used in Table 4a and explaining the difference between the two tables.

  2. The description of the training length is not consistent. The appendix states that all reported results use checkpoints from the first 100 updates, but in Figure 5 several methods reach the strict compliance thresholds after more than 100 steps, and the definition of a stable crossing also requires additional steps after the crossing. The authors may consider stating how many steps these runs were trained for.

  3. The claim that DARA learns more stably is not directly supported. The conclusion states that DARA learns faster and more stably, but the main text only mentions briefly that DARA-Asym converges more consistently across seeds. The main tables report only the mean over seeds for tool calling, and the mathematical reasoning comparisons use the same random seed at each scale, while the bootstrap confidence intervals mentioned in the appendix are not reported. Since many differences between methods are small, the authors may consider reporting the variation across seeds or the confidence intervals.

Questions for authors
  1. The appendix shows that the symmetric weighting scales the signal and the fluctuation of each reward by the same factor, so their ratio does not change, and the final batch normalization rescales all rewards by a common factor. Could the authors explain whether the faster learning comes from the change in the relative weights between rewards or from a change in the effective step size?

  2. In the later stage of training, the weight of the length reward reaches the cap, and DARA then uses the same weights as the static variant in Table 4a. Could the authors explain whether the difference between DARA and the static variant mainly comes from the early stage of training?

  3. DARA reaches the target behaviors earlier, but after longer training its final performance is comparable to the baselines. Could the authors explain how the practical benefit of DARA should be understood when the training budget is fixed and sufficiently long?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yYZL (1/3)

Thank you for the positive assessment and the careful questions.

"The results in Table 4a are not consistent with those in Table 3. [...] neither the caption nor the text states which checkpoint Table 4a is taken from."

The two tables report different checkpoints. Table 3 is the only table taken at step 50, as its caption states. All other tables use the step-100 checkpoint. This includes Table 4 and matches the final checkpoint of Tables 1 and 2. The values differ accordingly. DARA-Asym on DeepSeek-R1-1.5B has 45.83 accuracy and 8.16% Exceed at step 50. At step 100 it has 48.03 accuracy and 4.21% Exceed.

"The appendix states that all reported results use checkpoints from the first 100 updates, but in Figure 5 several methods reach the strict compliance thresholds after more than 100 steps, and the definition of a stable crossing also requires additional steps after the crossing."

There is no inconsistency. The sentence in Appendix D.1 refers to the evaluated checkpoints in Tables 3 and 4 and Figure 6. All of them are taken from the first 100 updates. Figure 5 is computed from the training rewards of the same runs. These runs continued training beyond step 100. Every reported crossing is therefore followed by the full 20-step persistence window:

BackboneMethodT99T99 + 20Logged through
Qwen3-4B-InstructGDPO111131417
Qwen3-4B-InstructDARA-Asym6686123
Qwen3-4B-InstructDARA-Sym5979104
DeepSeek-R1-7BGDPO142162294
DeepSeek-R1-7BDARA-Asym5676105
DeepSeek-R1-7BDARA-Sym5070103

"The conclusion states that DARA learns faster and more stably, but the main text only mentions briefly that DARA-Asym converges more consistently across seeds. The main tables report only the mean over seeds for tool calling [...]"

The stability claim refers to how consistently the target behavior is acquired across training seeds. Figure 1(b) shows it through the interquartile range over five seeds. The GDPO band extends to about step 60 while the bands of both DARA variants stay narrow. The per-seed steps at which the training format reward first reaches 0.8 make this explicit:

MethodSeeds 0 / 1 / 2 / 4 / 5MedianRange
GRPO37 / 43 / 34 / 31 / 313431–43
GDPO18 / 19 / 75 / 16 / 651916–75
DARA-Asym15 / 14 / 13 / 13 / 161413–16
DARA-Sym14 / 16 / 15 / 13 / 291513–29

This is consistent with Figure 1 of Liu et al. (2026), which also shows unstable GDPO training. The final checkpoints show the same pattern. On the 1.5B model the standard deviation of the final Average Format over five seeds is 0.34 for DARA-Asym and 0.64 for DARA-Sym. It is 1.00 for GDPO and 1.36 for GD$^2$PO-Hard. It reaches 2.91 for DVAO and 5.18 for GRPO.

AuthorsOpenReview ↗

Response to Reviewer yYZL (2/3)

"The mathematical reasoning comparisons use the same random seed at each scale, while the bootstrap confidence intervals mentioned in the appendix are not reported."

Our seed protocol follows the standard practice of this line of work. GDPO (Liu et al. (2026)) and GD$^2$PO (Liu et al. (2026b)) report five runs for tool calling and a single run per model for mathematical reasoning. DVAO (Jiang et al. (2026)) and DAPO (Yu et al. (2025)) report single runs. We use the same protocol: five seeds for tool calling and one seed per backbone for mathematical reasoning. Training variability is measured in tool calling as shown above. A single math run takes 21.7 to 190.8 hours on 8 H100 or H200 GPUs.

Instead of repeating one backbone, we replicate the math comparison across three backbones from two model families. DARA reaches the 99% threshold before GDPO on each of them (Figure 5 and the table above). On held-out benchmarks both DARA variants reach 95% length compliance at step 50 and GDPO only at step 70 (Figure 6).

The 95% bootstrap confidence intervals are listed below. They come from a step-50 re-evaluation with 2,000 question-level resamples:

SizeMethod (step 50)Acc (95% CI)Exceed (95% CI)Joint (95% CI)
1.5BGRPO45.71 [42.87, 48.62]10.82 [8.57, 13.06]44.66 [41.90, 47.54]
1.5BGDPO44.24 [41.35, 47.37]8.58 [6.69, 10.51]43.54 [40.73, 46.48]
1.5BDVAO45.84 [43.14, 48.69]10.82 [8.61, 13.00]44.85 [42.14, 47.77]
1.5BGD$^2$PO-Hard45.17 [42.45, 48.05]11.53 [9.42, 13.56]44.12 [41.46, 46.85]
1.5BDARA-Asym45.62 [42.68, 48.66]8.67 [6.83, 10.68]44.72 [41.79, 47.75]
1.5BDARA-Sym44.72 [41.80, 47.97]5.57 [4.18, 7.08]44.35 [41.40, 47.31]
4BDVAO60.12 [56.43, 63.83]5.09 [3.44, 6.95]59.56 [55.93, 63.20]
4BGD$^2$PO-Hard61.42 [57.75, 65.13]7.80 [5.51, 10.41]59.70 [56.06, 63.35]
4BDARA-Asym60.24 [56.61, 63.85]3.97 [2.46, 5.76]59.59 [55.90, 63.42]
4BDARA-Sym59.07 [55.51, 63.15]1.96 [1.18, 2.79]58.81 [55.08, 62.56]
7BDVAO57.89 [54.64, 61.25]6.13 [4.40, 7.90]57.09 [53.76, 60.48]
7BGD$^2$PO-Hard58.03 [54.82, 61.38]8.52 [6.48, 10.65]56.72 [53.50, 60.02]
7BDARA-Asym58.01 [54.92, 61.43]3.96 [2.65, 5.31]57.30 [54.09, 60.53]
7BDARA-Sym55.90 [52.69, 59.38]0.87 [0.59, 1.18]55.78 [52.66, 59.07]

DARA-Sym has the lowest Exceed at every scale.

"The appendix shows that the symmetric weighting scales the signal and the fluctuation of each reward by the same factor [...] and the final batch normalization rescales all rewards by a common factor. Could the authors explain whether the faster learning comes from the change in the relative weights between rewards or from a change in the effective step size?"

It comes from the relative weights. DARA cannot enlarge the step because the weighted advantage passes through the same final batch normalization as in GDPO. Whatever weights DARA assigns, this normalization rescales the weighted sum to unit variance. The total advantage scale of DARA and GDPO is thus identical at every step. In the tool-calling logs the advantage standard deviation is 1.000 at every step for GDPO and both DARA variants. The weights therefore survive only as relative magnitudes between rewards (Appendix A.3). What DARA changes is how the update is divided among the rewards, not its size.

AuthorsOpenReview ↗

Response to Reviewer yYZL (3/3)

"In the later stage of training, the weight of the length reward reaches the cap, and DARA then uses the same weights as the static variant in Table 4a. Could the authors explain whether the difference between DARA and the static variant mainly comes from the early stage of training?"

The difference comes from the phase in which the length reward is still active in many groups. This is the phase where adapting the weight matters. There DARA gives length a weight well below 5. The static variant applies 5 from the first step. It overweights length and loses accuracy (48.03 → 46.84 for the asymmetric variant in Table 4(a)).

For DARA-Asym this phase covers the whole evaluated window. Its length weight never reaches the cap within the 100 evaluated updates on DeepSeek-R1-1.5B. It stays near 1.00–1.18 during steps 1–40 while the length density is 0.60–0.83. It then rises to 2.31 at step 60 and to 3.57 at step 100 as the density falls toward 0.08. DARA-Sym first reaches the cap at step 59 (Appendix A.2).

The intermediate fixed weights we add for Reviewer s7D4 confirm that a constant cannot replace this schedule. Static-Asym with w = 3 matches the speed of DARA-Asym (T99 at step 67 vs. 68). Its accuracy is lower at step 50 (44.41% vs. 45.83%) and at step 100 (47.73% vs. 48.03%). Static-Asym with w = 2 is slower and reaches T99 only at step 73.

In tool calling the cap rarely binds. The format weight follows format density throughout training. It is about 3 at the start when few responses follow the format. It drops to 1.1–1.9 around step 10 when format is active in the most groups. It rises to 2.3–3.0 as format saturates. It is at the cap in only about 1% of the steps after step 40. A static weight cannot follow this trajectory.

"DARA reaches the target behaviors earlier, but after longer training its final performance is comparable to the baselines. Could the authors explain how the practical benefit of DARA should be understood when the training budget is fixed and sufficiently long?"

The practical benefit holds even with a long budget. There are three reasons.

  • Lower cost to reach the target. Tool-calling methods have the same cost per step (85.2 s for GDPO and 85.0 s for DARA-Asym). DARA therefore reaches format reward 0.8 in 26% less time at the median seed. Averaged over seeds it needs 63% less time (20.1 min vs. 54.8 min). The saving matters most where training is expensive. Our math comparison alone used 18,156.6 GPU-hours. There DARA-Asym reaches the 99% threshold after 25.0 hours of training against 41.5 for GDPO on Qwen3-4B-Instruct. On DeepSeek-R1-7B it needs 28.3 hours against 62.3 (see our reply to Reviewer s7D4).
  • A predictable convergence time. A budget is fixed before training without knowing which seed will be slow. To cover every seed in the table above GDPO needs a budget of 75 steps to reach the target. DARA-Asym needs 16.
  • No loss at the end of training. At step 100 both DARA variants are at or above GRPO and GDPO in Average Accuracy and Average Format on the 1.5B tool-calling model (Table 1). In mathematical reasoning at step 100 DARA-Asym stays within 0.8 points of the accuracy of GRPO. It also produces fewer over-length responses (0.25% vs. 0.70% on Qwen3-4B-Instruct in Table 4(b)).

Reference:

  • Liu et al. (2026). "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization." ICML, 2026.
  • Liu et al. (2026b). "GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization." Preprint, 2026.
  • Jiang et al. (2026). "DVAO: Dynamic Variance-Adaptive Advantage Optimization for Multi-Reward Reinforcement Learning." Preprint, 2026.
  • Yu et al. (2025). "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS, 2025.
AuthorsOpenReview ↗

Follow-up: Brief Summary of Our Responses

Dear Reviewer yYZL,

Thank you again for your positive assessment. As the discussion period is about to end, we briefly summarize our responses.

Table 3 is the only table at step 50 and all other tables use step 100. The first-100-updates statement in Appendix D.1 refers to the evaluated checkpoints. The runs behind Figure 5 continued beyond step 100. The per-seed results show that GDPO is unstable across seeds, which matches Figure 1 of the GDPO paper. For mathematical reasoning we follow the protocol of GDPO and GD²PO and report bootstrap confidence intervals.

The gain of DARA comes from the relative weights because the final batch normalization fixes the total advantage scale. DARA-Asym never reaches the weight cap within the 100 evaluated updates. With a long budget DARA still reaches the target at lower cost and with a predictable convergence time.

If these responses address your concerns, we would be grateful if you could consider updating your score. We are happy to answer any remaining questions.

Best regards,

The Authors

Reviewer_7kuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates uneven learning signals across reward objectives in multi-reward reinforcement learning. The authors identify an imbalance in advantage energy caused by differences in active-group density, even after reward-wise normalization in GDPO. To address this issue, they propose DARA, which dynamically adjusts advantage weights based on reward activity to strengthen sparse reward signals. Experiments on tool calling and mathematical reasoning demonstrate faster learning of format and length constraints while maintaining competitive final performance.

Critical strengths
  1. The paper identifies an important issue in multi-reward RL: uneven learning signals across reward objectives, even after reward-wise normalization.

  2. DARA introduces a lightweight, adaptive reward reweighting strategy that is easy to implement without modifying the underlying RL objective.

  3. Experiments across multiple models and tasks demonstrate faster learning of sparse reward objectives, particularly format and length compliance.

Critical weaknesses
  1. My primary concern lies in the practical significance of the proposed method. Although Figure 2 demonstrates that DARA substantially accelerates the learning of format compliance, this improvement does not appear to translate into faster convergence or meaningful gains in tool-calling accuracy, which is arguably the more important metric in practice. In fact, several baseline methods appear to exhibit comparable or even more favorable accuracy convergence trends. This raises concerns about whether DARA genuinely improves multi-reward optimization or merely accelerates the learning of an auxiliary objective.

  2. The proposed method essentially assigns larger weights to rewards with lower active-group density. However, it is unclear whether lower density necessarily implies that a reward deserves greater emphasis. For instance, a reward may have low density because it is already nearly saturated, rather than because it is insufficiently optimized. Blindly amplifying such rewards could introduce unnecessary optimization pressure, exacerbate conflicts between objectives, and potentially compromise performance on the primary task. The paper lacks sufficient analysis of these potential negative effects and a clear justification for why equalizing advantage energy should lead to better overall optimization.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7kuu (1/2)

Thank you for the review.

"Although Figure 2 demonstrates that DARA substantially accelerates the learning of format compliance, this improvement does not appear to translate into faster convergence or meaningful gains in tool-calling accuracy, which is arguably the more important metric in practice. [...] This raises concerns about whether DARA genuinely improves multi-reward optimization or merely accelerates the learning of an auxiliary objective."

Format is not an auxiliary objective in this task. It is one of the two rewards the policy is trained on. It encodes the output structure that the tool runtime parses. BFCL's accuracy checker scores the extracted function calls whether or not the output follows this structure. Accuracy and Format therefore measure two different requirements. Figure 2 shows this directly. At step 20 GDPO already reaches accuracy close to its final level while its format compliance is still below 50%. A tool-calling model is usable only when both requirements hold. Multi-reward RL is the setting where both are optimized together.

The overlapping accuracy curves are what DARA is designed to produce. Correctness is the most active reward throughout training. DARA therefore keeps its weight at 1 and leaves its learning speed unchanged. This holds in every step of the DARA-Asym runs and in 97% of the steps of the DARA-Sym runs. What DARA changes is the sparse objective. Format no longer lags behind correctness. The improvement is in reaching both objectives together:

  • Under a matched budget of 60 steps DARA-Sym and DARA-Asym have the highest Average Accuracy (50.94% and 50.69%) and the highest Average Format (96.06% and 96.02%). The baselines reach at most 50.50% and 90.11% (Sec. 4.1).
  • At step 100 both variants improve Average Accuracy and Average Format over GRPO and GDPO on the 1.5B model (Table 1).
  • DARA is also more consistent across seeds. Two of the five GDPO seeds need 65 and 75 steps to reach a training format reward of 0.8. All five DARA-Asym seeds reach it within steps 13–16. On the 1.5B model the standard deviation of the final Average Format over five seeds is 0.34 for DARA-Asym versus 1.00 for GDPO.
  • In mathematical reasoning the Joint metric requires a response to be both correct and within the length limit. DARA-Asym has the highest Joint on both DeepSeek-R1 models (Table 3).
  • Our new fixed-weight sweep on DeepSeek-R1-1.5B points the same way. Static-Asym with w = 3 matches DARA-Asym in training-side T99 (67 vs. 68 steps). DARA-Asym still has higher Acc and Joint at step 50 (45.83%/45.14% vs. 44.41%/44.04%) and at step 100 (48.03%/47.76% vs. 47.73%/47.43%).
AuthorsOpenReview ↗

Response to Reviewer 7kuu (2/2)

"However, it is unclear whether lower density necessarily implies that a reward deserves greater emphasis. For instance, a reward may have low density because it is already nearly saturated, rather than because it is insufficiently optimized. Blindly amplifying such rewards could introduce unnecessary optimization pressure, exacerbate conflicts between objectives, and potentially compromise performance on the primary task. The paper lacks sufficient analysis of these potential negative effects and a clear justification for why equalizing advantage energy should lead to better overall optimization."

DARA does not amplify a saturated reward blindly. There are three reasons.

First, the weight applies only to groups where the reward is still violated. A group in which all responses already satisfy the reward has zero advantage for that reward. This holds whatever its weight is. The weight multiplies only the mixed groups where some responses satisfy the reward and others do not. Near saturation these groups are exactly the remaining failures. DARA therefore adds no pressure on behavior that has already been learned.

Second, amplification restores parity and never creates dominance. The weight makes the calibrated energy of a sparse reward equal to that of the most active reward: wk2Ek=min⁡{wmax⁡2Ek, Eref}w_k^2E_k=\min\{w_{\max}^2E_k,\,E_{\mathrm{ref}}\} (corollary in Appendix A.3). A nearly saturated reward thus contributes at most as much batch-level signal as the primary reward. The cap wmax⁡w_{\max} also bounds the per-group amplification.

Third, the training logs show no cost to the primary objective. By step 100 the format density of the DARA runs falls to 1.5% and the format weight rises to about 3. BFCL accuracy at step 100 is still on par with or higher than that of GDPO. On the 1.5B model it is 51.17 for DARA-Sym and 50.59 for DARA-Asym against 50.37. On the 3B model it is 54.15 and 53.63 against 53.66. Mathematical reasoning shows the same. The length weight of DARA-Asym stays near 1.00–1.18 during steps 1–40 while the length density is 0.60–0.83. It rises to 3.57 by step 100 as the density approaches 0.08. Its step-100 accuracy is still the highest in our fixed-weight sweep. Blind amplification looks different. Fixing the length weight at 5 from the start reaches T99 earlier (62 vs. 68 steps) but drops accuracy from 48.03 to 46.84 (Table 4(a)).

On why equal energy is the right target: GDPO normalizes each reward within its group so that all rewards enter the update on equal footing. Our analysis shows that this equal footing holds within a group but not across the batch. A reward active in fewer groups contributes proportionally less energy (4 : 1 in the example of Figure 1(a)). DARA restores at the batch level the balance that GDPO intends at the group level. Appendix A.3 further shows that every active group of a binary reward pushes the policy toward raising the success probability of that reward on its prompt in expectation. The energy of a reward therefore counts the contributions that improve it.

AuthorsOpenReview ↗

Follow-up: Brief Summary of Our Responses

Dear Reviewer 7kuu,

Thank you again for your review. As the discussion period is about to end, we briefly summarize our responses.

Format is one of the two training rewards and encodes the structure that the tool runtime parses. A tool-calling model is usable only when both accuracy and format hold. DARA keeps the correctness weight at 1 and reaches the highest accuracy and format compliance under a matched 60-step budget.

DARA does not amplify a saturated reward blindly. Its weight acts only on groups that still fail. It restores at the batch level the balance that GDPO intends within each group.

If these responses address your concerns, we would be grateful if you could consider updating your score. We are happy to answer any remaining questions.

Best regards, The Authors

Reviewer_s7D4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies imbalance between reward channels after reward-wise group normalization. Under idealized GDPO normalization, a reward's squared-advantage energy is proportional to its fraction of active groups. DARA compensates using capped inverse-square-root density weights, either symmetrically or by amplifying positive advantages. Experiments on tool calling and mathematical reasoning report earlier acquisition of format or length constraints with competitive final performance. The contribution is a simple adaptive aggregation rule and an empirical study of constraint acquisition; the energy identity itself does not guarantee faster policy improvement.

Critical strengths
  1. A concrete and inexpensive correction. Active-group density identifies a batch-level imbalance that reward-wise normalization leaves unresolved. The density–energy identity is straightforward under the stated assumptions, and the correction adds little computation. The paper appropriately distinguishes exact energy matching for uncapped DARA-Sym from the asymmetric variant. This makes the method practical and its basic motivation understandable.

  2. Useful empirical breadth and outcome reporting. The study covers tool calling and mathematical reasoning, several backbones, a three-reward setting, and both variants. The tool-calling results use five training seeds, and Appendix C includes group-size comparisons at a fixed response budget. Reporting accuracy, compliance, and joint success helps expose trade-offs rather than equating shorter outputs with better reasoning. These experiments provide credible evidence of faster early constraint acquisition.

Critical weaknesses
  1. The training protocol behind the main efficiency claim needs reconciliation. Section 4.2 reports GDPO reaching the 99% compliance threshold at steps 111 and 142, whereas Appendix D.1 (p. 21) says all reported results use checkpoints from the first 100 updates. The crossing definition additionally requires 20 subsequent steps above threshold. Extended training logs may resolve this, but their scope is not clear. The number of independent math training seeds is also unclear; question-bootstrap intervals do not measure training variability. These issues affect confidence in the claimed reduction of up to 65% in training steps. With dynamic filtering and changing output lengths, step savings also need to be distinguished from generated-token or elapsed-time savings.

  2. The evidence does not yet isolate the benefit of density adaptation over a simpler reward trade-off. Table 4(a) tests a fixed length weight of 5, but no intermediate fixed-weight sweep. This matters because accuracy improvements are small or model dependent: on Qwen3-4B at step 50, DARA-Asym gives 60.30 accuracy and 59.68 joint success, versus 62.14 and 60.48 for GRPO (Table 3). Appendix A.3 connects advantage energy to gradient fluctuations under additional assumptions, not necessarily to useful mean gradients or faster learning. I therefore need stronger empirical separation of adaptive weighting from merely emphasizing the length reward.

Questions for authors
  1. Which runs and stopping horizons produced Figure 5, including crossings beyond step 100 and their 20-step persistence? Does the first-100-updates statement apply only to held-out checkpoint evaluation? Please specify the math training seeds and provide per-seed crossings and generated-token counts, including rejected groups, for the headline comparison.

  2. Does DARA improve the accuracy–compliance frontier over fixed weights between 1 and 5 under matched budgets? A small sweep, including comparison at matched compliance, would directly test whether density adaptation explains the gain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer s7D4 (1/3)

Thank you for the detailed review.

"Section 4.2 reports GDPO reaching the 99% compliance threshold at steps 111 and 142, whereas Appendix D.1 (p. 21) says all reported results use checkpoints from the first 100 updates. [...] Which runs and stopping horizons produced Figure 5, including crossings beyond step 100 and their 20-step persistence? Does the first-100-updates statement apply only to held-out checkpoint evaluation?"

There is no inconsistency. The sentence in Appendix D.1 refers to the evaluated checkpoints in Tables 3 and 4 and Figure 6. All of them are taken from the first 100 updates. Figure 5 is computed from the training rewards of the same runs. These runs continued training beyond step 100. Every reported crossing is therefore followed by the full 20-step persistence window:

BackboneMethodT99T99 + 20Logged through
Qwen3-4B-InstructGDPO111131417
Qwen3-4B-InstructDARA-Asym6686123
Qwen3-4B-InstructDARA-Sym5979104
DeepSeek-R1-7BGDPO142162294
DeepSeek-R1-7BDARA-Asym5676105
DeepSeek-R1-7BDARA-Sym5070103

"The number of independent math training seeds is also unclear; question-bootstrap intervals do not measure training variability. [...] Please specify the math training seeds and provide per-seed crossings and generated-token counts, including rejected groups, for the headline comparison."

Our seed protocol follows the standard practice of this line of work. GDPO (Liu et al. (2026)) and GD$^2$PO (Liu et al. (2026b)) report five runs for tool calling and a single run per model for mathematical reasoning. DVAO (Jiang et al. (2026)) and DAPO (Yu et al. (2025)) report single runs. We use the same protocol: five seeds for tool calling and one seed (seed 0) per backbone for mathematical reasoning. A single math run takes 21.7 to 190.8 hours on 8 H100 or H200 GPUs. The comparison of six methods on three backbones therefore already amounts to 18,156.6 GPU-hours.

Instead of repeating one backbone, we replicate the math comparison across three backbones from two model families. DARA reaches the 99% threshold before GDPO on each of them (Figure 5 and the tables in this reply). The same ordering appears on held-out benchmarks. Both DARA variants reach 95% length compliance at step 50 and GDPO only at step 70 (Figure 6).

Training variability is measured directly in tool calling. There every main result uses five independent seeds. The steps at which the training format reward first reaches 0.8 are:

MethodSeeds 0 / 1 / 2 / 4 / 5MedianRange
GRPO37 / 43 / 34 / 31 / 313431–43
GDPO18 / 19 / 75 / 16 / 651916–75
DARA-Asym15 / 14 / 13 / 13 / 161413–16
DARA-Sym14 / 16 / 15 / 13 / 291513–29

Two of the five GDPO seeds need 65 and 75 steps, while all DARA-Asym seeds cross within steps 13–16. This is consistent with Figure 1 of Liu et al. (2026), which also shows unstable GDPO training.

"With dynamic filtering and changing output lengths, step savings also need to be distinguished from generated-token or elapsed-time savings."

The step savings are also token and time savings. In tool calling every step samples a fixed 2,048 responses without dynamic filtering. On the same hardware the mean time per step is 85.2 s for GDPO and 85.0 s for DARA-Asym (86.1 s for DARA-Sym; five seeds × 100 steps each). DARA-Asym therefore reaches format reward 0.8 in 19.8 min at the median seed against 27.0 min for GDPO. Averaged over seeds the times are 20.1 min and 54.8 min.

In mathematical reasoning the token counts below include the responses later rejected by dynamic filtering. Each count sums the generated responses times their mean output length over all updates up to T99. Prompt tokens are excluded.

BackboneMethodT99Generated output tokensReduction vs. GDPOTraining time to T99
Qwen3-4B-InstructGDPO1115.061B—41.5 h
Qwen3-4B-InstructDARA-Asym663.208B36.6%25.0 h
Qwen3-4B-InstructDARA-Sym592.893B42.8%25.7 h
DeepSeek-R1-7BGDPO1426.859B—62.3 h
DeepSeek-R1-7BDARA-Asym563.059B55.4%28.3 h
DeepSeek-R1-7BDARA-Sym502.748B59.9%27.6 h
AuthorsOpenReview ↗

Response to Reviewer s7D4 (2/3)

"Table 4(a) tests a fixed length weight of 5, but no intermediate fixed-weight sweep. [...] Does DARA improve the accuracy–compliance frontier over fixed weights between 1 and 5 under matched budgets? A small sweep, including comparison at matched compliance, would directly test whether density adaptation explains the gain."

GDPO itself already runs this kind of sweep. It varies the fixed length weight on DeepSeek-R1-7B and finds that "simply adjusting reward weights does not reliably induce the intended prioritization" (Fig. 6 and Appendix G of Liu et al. (2026)). Such a sweep is also expensive. Every weight requires a full training run. In our math setting a single run takes up to 190.8 hours on 8 GPUs (see above). Avoiding this search is a practical advantage of DARA. The method is simple and effective. It derives the weight from the measured density of each batch and requires no weight tuning.

We nevertheless ran a small sweep to answer the question directly. On DeepSeek-R1-1.5B we train the static asymmetric variant with length weights 2, 3 and 5 under the same 100-step budget as DARA-Asym. A weight of 1 is GDPO.

DeepSeek-R1-1.5BT99 ↓Step 50 Acc ↑Step 50 Joint ↑Step 100 Acc ↑Step 100 Exceed ↓Step 100 Joint ↑
GDPO (w = 1)18544.1243.4747.053.9346.84
Static-Asym (w = 2)7344.0143.5647.013.3646.74
Static-Asym (w = 3)6744.4144.0447.734.7747.43
Static-Asym (w = 5)6243.8343.5646.843.1446.72
DARA-Asym6845.8345.1448.034.2147.76

DARA-Asym has the highest accuracy and Joint of all weights at both checkpoints. The comparison at matched compliance speed is w = 3. It reaches the 99% threshold at step 67 against 68 for DARA-Asym. At this matched speed DARA-Asym is 1.42 points more accurate at step 50. At step 100 it also has higher accuracy and Joint and lower Exceed. The largest fixed weight w = 5 reaches the threshold earliest and has the lowest Exceed. It pays for this with lower accuracy. The symmetric variant shows the same cost of a fixed weight in Table 4(a) (46.71 → 45.97 accuracy).

DARA avoids this trade-off because its weight follows the density of the length reward. The length weight of DARA-Asym stays near 1.00–1.18 during steps 1–40 while length is active in most groups. It then rises to 2.31 at step 60 and to 3.57 at step 100 as this density falls. In tool calling the format weight even falls from about 3 to 1.1–1.9 around step 10 and rises again to 2.3–3.0 as format saturates. No constant follows such a trajectory.

"This matters because accuracy improvements are small or model dependent: on Qwen3-4B at step 50, DARA-Asym gives 60.30 accuracy and 59.68 joint success, versus 62.14 and 60.48 for GRPO (Table 3)."

DARA is designed to acquire sparse objectives faster without hurting the dense ones. Its accuracy should be judged against that goal. On both DeepSeek-R1 models DARA-Asym has the highest accuracy and the highest Joint at step 50 (45.83 / 45.14 and 58.31 / 57.83). On Qwen3-4B-Instruct at step 50 GRPO still produces 8.07% over-length responses against 4.38% for DARA-Asym. Enforcing the length constraint first lowers accuracy for every method before it recovers (Figure 6). By step 100 length compliance has converged for all methods. The accuracy gap then shrinks to 62.17 versus 62.99 while DARA-Asym lowers Exceed from 0.70% to 0.25% (Table 4(b)). Qwen3-4B-Thinking shows the same pattern with 64.23 vs. 64.82 accuracy and 0.94% vs. 1.86% Exceed.

AuthorsOpenReview ↗

Response to Reviewer s7D4 (3/3)

"Appendix A.3 connects advantage energy to gradient fluctuations under additional assumptions, not necessarily to useful mean gradients or faster learning."

Appendix A.3 also characterizes the mean gradient. The proposition on the signal direction of a binary reward averages over the other rewards. Every active group of a binary reward then contributes a positive multiple of ∇θpi,k\nabla_\theta p_{i,k}. This is the gradient that raises the success probability of that reward on its prompt. The positive-only term of DARA-Asym contributes a positive multiple of the same gradient. Inactive groups contribute zero. The energy of a reward therefore counts exactly the groups whose expected contribution improves that reward. DARA scales up only those contributions.

Convergence-rate bounds are not how methods in this area are evaluated. GRPO and GDPO are likewise justified by their mechanism and experiments rather than by convergence rates. Our analysis explains the uneven learning that GDPO itself reports. It also makes quantitative predictions that the experiments confirm:

  • Density drives learning speed. At a fixed budget of 2,048 responses per step a larger group size raises format density. Raising the group size from 4 to 32 makes format reach 0.8 earlier for every method (Figure 3).
  • Methods separate where energy is most unequal. Under GDPO normalization the length-to-correctness energy ratio falls below 0.04 late in training. DARA keeps it at 1 until the cap binds and at a median of 0.39–0.49 afterwards (Appendix A.2). This late phase is where DARA and GDPO separate most (Figure 5).
  • Correcting the energy brings learning forward. DARA moves both the density peak and the rapid-learning phase of format earlier (Figure 1(b)). This holds on every seed (table above).

Advantage energy thus predicts which objective learns slowly and when. Correcting it makes that objective learn faster.

Reference:

  • Liu et al. (2026). "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization." ICML, 2026.
  • Liu et al. (2026b). "GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization." Preprint, 2026.
  • Jiang et al. (2026). "DVAO: Dynamic Variance-Adaptive Advantage Optimization for Multi-Reward Reinforcement Learning." Preprint, 2026.
  • Yu et al. (2025). "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS, 2025.
AuthorsOpenReview ↗

Follow-up: Brief Summary of Our Responses

Dear Reviewer s7D4,

Thank you again for your detailed review. As the discussion period is about to end, we briefly summarize our responses.

The first-100-updates statement in Appendix D.1 refers to the evaluated checkpoints. The runs behind Figure 5 continued beyond step 100. Math experiments are expensive and this comparison alone used 18,156.6 GPU-hours. We therefore follow the single-run protocol of GDPO and GD²PO and replicate the comparison across three backbones. The step savings of DARA are also savings in generated tokens and training time.

GDPO itself found that adjusting fixed weights is unreliable. In our new sweep DARA-Asym has the highest accuracy and Joint without any weight tuning. On both DeepSeek-R1 models it also has the highest accuracy and Joint at step 50. Our analysis explains the uneven learning reported by GDPO and directly guides the design of DARA.

If these responses address your concerns, we would be grateful if you could consider updating your score. We are happy to answer any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.