- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors focus on multi-reward RL for LLMs, and in particular on the uneven learning progress across objectives that remains after GDPO normalizes each reward separately. To address this problem, the authors first show that, under idealized normalization, the total advantage signal of a reward over a batch is proportional to the fraction of groups in which this reward actually varies, so rewards that vary in fewer groups contribute less to training. Based on this relation, the authors propose DARA, which computes a weight for each reward from this fraction on every batch and gives larger weights to rewards that vary in fewer groups. Experiments on tool calling and mathematical reasoning show that DARA reaches the target behaviors in fewer training steps than the baselines, while remaining competitive in final performance.
Critical strengths
-
In this paper, the authors start from the uneven learning reported in GDPO and ask what causes it and whether reward weights can be derived from this cause, so the motivation is clear. The authors also give clear definitions of the key quantities, and the main relation can be derived in a few lines, which makes the core idea of the paper easy to follow.
-
The authors provide sufficient details for reproduction, including the training and evaluation configurations, the reward definitions, the pseudocode, and the definitions of the key metrics, such as the over-length rate and the joint success rate that requires both correctness and length compliance.
-
The authors present the method and the results with good visualizations. Figure 1 uses a small example to show how the density, the weights and the energy ratio are computed, together with the training curves of the format reward and its density. The figures on the group size, on the training step at which each compliance level is reached, and on the energy ratio during training further connect the analysis with the experimental results. This makes the paper easy to catch up on.
Critical weaknesses
-
The results in Table 4a are not consistent with those in Table 3. Both tables report DARA-Asym and DARA-Sym on DeepSeek-R1-1.5B, but the accuracy and the over-length rate in Table 4a are clearly different from those in Table 3, and neither the caption nor the text states which checkpoint Table 4a is taken from. The authors may consider stating the training step used in Table 4a and explaining the difference between the two tables.
-
The description of the training length is not consistent. The appendix states that all reported results use checkpoints from the first 100 updates, but in Figure 5 several methods reach the strict compliance thresholds after more than 100 steps, and the definition of a stable crossing also requires additional steps after the crossing. The authors may consider stating how many steps these runs were trained for.
-
The claim that DARA learns more stably is not directly supported. The conclusion states that DARA learns faster and more stably, but the main text only mentions briefly that DARA-Asym converges more consistently across seeds. The main tables report only the mean over seeds for tool calling, and the mathematical reasoning comparisons use the same random seed at each scale, while the bootstrap confidence intervals mentioned in the appendix are not reported. Since many differences between methods are small, the authors may consider reporting the variation across seeds or the confidence intervals.
Questions for authors
-
The appendix shows that the symmetric weighting scales the signal and the fluctuation of each reward by the same factor, so their ratio does not change, and the final batch normalization rescales all rewards by a common factor. Could the authors explain whether the faster learning comes from the change in the relative weights between rewards or from a change in the effective step size?
-
In the later stage of training, the weight of the length reward reaches the cap, and DARA then uses the same weights as the static variant in Table 4a. Could the authors explain whether the difference between DARA and the static variant mainly comes from the early stage of training?
-
DARA reaches the target behaviors earlier, but after longer training its final performance is comparable to the baselines. Could the authors explain how the practical benefit of DARA should be understood when the training budget is fixed and sufficiently long?
4 comments
Response to Reviewer yYZL (2/3)
"The mathematical reasoning comparisons use the same random seed at each scale, while the bootstrap confidence intervals mentioned in the appendix are not reported."
Our seed protocol follows the standard practice of this line of work. GDPO (Liu et al. (2026)) and GD$^2$PO (Liu et al. (2026b)) report five runs for tool calling and a single run per model for mathematical reasoning. DVAO (Jiang et al. (2026)) and DAPO (Yu et al. (2025)) report single runs. We use the same protocol: five seeds for tool calling and one seed per backbone for mathematical reasoning. Training variability is measured in tool calling as shown above. A single math run takes 21.7 to 190.8 hours on 8 H100 or H200 GPUs.
Instead of repeating one backbone, we replicate the math comparison across three backbones from two model families. DARA reaches the 99% threshold before GDPO on each of them (Figure 5 and the table above). On held-out benchmarks both DARA variants reach 95% length compliance at step 50 and GDPO only at step 70 (Figure 6).
The 95% bootstrap confidence intervals are listed below. They come from a step-50 re-evaluation with 2,000 question-level resamples:
| Size | Method (step 50) | Acc (95% CI) | Exceed (95% CI) | Joint (95% CI) |
|---|---|---|---|---|
| 1.5B | GRPO | 45.71 [42.87, 48.62] | 10.82 [8.57, 13.06] | 44.66 [41.90, 47.54] |
| 1.5B | GDPO | 44.24 [41.35, 47.37] | 8.58 [6.69, 10.51] | 43.54 [40.73, 46.48] |
| 1.5B | DVAO | 45.84 [43.14, 48.69] | 10.82 [8.61, 13.00] | 44.85 [42.14, 47.77] |
| 1.5B | GD$^2$PO-Hard | 45.17 [42.45, 48.05] | 11.53 [9.42, 13.56] | 44.12 [41.46, 46.85] |
| 1.5B | DARA-Asym | 45.62 [42.68, 48.66] | 8.67 [6.83, 10.68] | 44.72 [41.79, 47.75] |
| 1.5B | DARA-Sym | 44.72 [41.80, 47.97] | 5.57 [4.18, 7.08] | 44.35 [41.40, 47.31] |
| 4B | DVAO | 60.12 [56.43, 63.83] | 5.09 [3.44, 6.95] | 59.56 [55.93, 63.20] |
| 4B | GD$^2$PO-Hard | 61.42 [57.75, 65.13] | 7.80 [5.51, 10.41] | 59.70 [56.06, 63.35] |
| 4B | DARA-Asym | 60.24 [56.61, 63.85] | 3.97 [2.46, 5.76] | 59.59 [55.90, 63.42] |
| 4B | DARA-Sym | 59.07 [55.51, 63.15] | 1.96 [1.18, 2.79] | 58.81 [55.08, 62.56] |
| 7B | DVAO | 57.89 [54.64, 61.25] | 6.13 [4.40, 7.90] | 57.09 [53.76, 60.48] |
| 7B | GD$^2$PO-Hard | 58.03 [54.82, 61.38] | 8.52 [6.48, 10.65] | 56.72 [53.50, 60.02] |
| 7B | DARA-Asym | 58.01 [54.92, 61.43] | 3.96 [2.65, 5.31] | 57.30 [54.09, 60.53] |
| 7B | DARA-Sym | 55.90 [52.69, 59.38] | 0.87 [0.59, 1.18] | 55.78 [52.66, 59.07] |
DARA-Sym has the lowest Exceed at every scale.
"The appendix shows that the symmetric weighting scales the signal and the fluctuation of each reward by the same factor [...] and the final batch normalization rescales all rewards by a common factor. Could the authors explain whether the faster learning comes from the change in the relative weights between rewards or from a change in the effective step size?"
It comes from the relative weights. DARA cannot enlarge the step because the weighted advantage passes through the same final batch normalization as in GDPO. Whatever weights DARA assigns, this normalization rescales the weighted sum to unit variance. The total advantage scale of DARA and GDPO is thus identical at every step. In the tool-calling logs the advantage standard deviation is 1.000 at every step for GDPO and both DARA variants. The weights therefore survive only as relative magnitudes between rewards (Appendix A.3). What DARA changes is how the update is divided among the rewards, not its size.
Response to Reviewer yYZL (3/3)
"In the later stage of training, the weight of the length reward reaches the cap, and DARA then uses the same weights as the static variant in Table 4a. Could the authors explain whether the difference between DARA and the static variant mainly comes from the early stage of training?"
The difference comes from the phase in which the length reward is still active in many groups. This is the phase where adapting the weight matters. There DARA gives length a weight well below 5. The static variant applies 5 from the first step. It overweights length and loses accuracy (48.03 → 46.84 for the asymmetric variant in Table 4(a)).
For DARA-Asym this phase covers the whole evaluated window. Its length weight never reaches the cap within the 100 evaluated updates on DeepSeek-R1-1.5B. It stays near 1.00–1.18 during steps 1–40 while the length density is 0.60–0.83. It then rises to 2.31 at step 60 and to 3.57 at step 100 as the density falls toward 0.08. DARA-Sym first reaches the cap at step 59 (Appendix A.2).
The intermediate fixed weights we add for Reviewer s7D4 confirm that a constant cannot replace this schedule. Static-Asym with w = 3 matches the speed of DARA-Asym (T99 at step 67 vs. 68). Its accuracy is lower at step 50 (44.41% vs. 45.83%) and at step 100 (47.73% vs. 48.03%). Static-Asym with w = 2 is slower and reaches T99 only at step 73.
In tool calling the cap rarely binds. The format weight follows format density throughout training. It is about 3 at the start when few responses follow the format. It drops to 1.1–1.9 around step 10 when format is active in the most groups. It rises to 2.3–3.0 as format saturates. It is at the cap in only about 1% of the steps after step 40. A static weight cannot follow this trajectory.
"DARA reaches the target behaviors earlier, but after longer training its final performance is comparable to the baselines. Could the authors explain how the practical benefit of DARA should be understood when the training budget is fixed and sufficiently long?"
The practical benefit holds even with a long budget. There are three reasons.
- Lower cost to reach the target. Tool-calling methods have the same cost per step (85.2 s for GDPO and 85.0 s for DARA-Asym). DARA therefore reaches format reward 0.8 in 26% less time at the median seed. Averaged over seeds it needs 63% less time (20.1 min vs. 54.8 min). The saving matters most where training is expensive. Our math comparison alone used 18,156.6 GPU-hours. There DARA-Asym reaches the 99% threshold after 25.0 hours of training against 41.5 for GDPO on Qwen3-4B-Instruct. On DeepSeek-R1-7B it needs 28.3 hours against 62.3 (see our reply to Reviewer s7D4).
- A predictable convergence time. A budget is fixed before training without knowing which seed will be slow. To cover every seed in the table above GDPO needs a budget of 75 steps to reach the target. DARA-Asym needs 16.
- No loss at the end of training. At step 100 both DARA variants are at or above GRPO and GDPO in Average Accuracy and Average Format on the 1.5B tool-calling model (
Table 1). In mathematical reasoning at step 100 DARA-Asym stays within 0.8 points of the accuracy of GRPO. It also produces fewer over-length responses (0.25% vs. 0.70% on Qwen3-4B-Instruct inTable 4(b)).
Reference:
Liu et al. (2026). "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization." ICML, 2026.Liu et al. (2026b). "GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization." Preprint, 2026.Jiang et al. (2026). "DVAO: Dynamic Variance-Adaptive Advantage Optimization for Multi-Reward Reinforcement Learning." Preprint, 2026.Yu et al. (2025). "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS, 2025.
Follow-up: Brief Summary of Our Responses
Dear Reviewer yYZL,
Thank you again for your positive assessment. As the discussion period is about to end, we briefly summarize our responses.
Table 3 is the only table at step 50 and all other tables use step 100. The first-100-updates statement in Appendix D.1 refers to the evaluated checkpoints. The runs behind Figure 5 continued beyond step 100. The per-seed results show that GDPO is unstable across seeds, which matches Figure 1 of the GDPO paper. For mathematical reasoning we follow the protocol of GDPO and GD²PO and report bootstrap confidence intervals.
The gain of DARA comes from the relative weights because the final batch normalization fixes the total advantage scale. DARA-Asym never reaches the weight cap within the 100 evaluated updates. With a long budget DARA still reaches the target at lower cost and with a predictable convergence time.
If these responses address your concerns, we would be grateful if you could consider updating your score. We are happy to answer any remaining questions.
Best regards,
The Authors
Response to Reviewer yYZL (1/3)
Thank you for the positive assessment and the careful questions.
The two tables report different checkpoints.
Table 3is the only table taken at step 50, as its caption states. All other tables use the step-100 checkpoint. This includesTable 4and matches the final checkpoint ofTables 1and2. The values differ accordingly. DARA-Asym on DeepSeek-R1-1.5B has 45.83 accuracy and 8.16% Exceed at step 50. At step 100 it has 48.03 accuracy and 4.21% Exceed.There is no inconsistency. The sentence in
Appendix D.1refers to the evaluated checkpoints inTables 3and4andFigure 6. All of them are taken from the first 100 updates.Figure 5is computed from the training rewards of the same runs. These runs continued training beyond step 100. Every reported crossing is therefore followed by the full 20-step persistence window:The stability claim refers to how consistently the target behavior is acquired across training seeds.
Figure 1(b)shows it through the interquartile range over five seeds. The GDPO band extends to about step 60 while the bands of both DARA variants stay narrow. The per-seed steps at which the training format reward first reaches 0.8 make this explicit:This is consistent with Figure 1 of
Liu et al. (2026), which also shows unstable GDPO training. The final checkpoints show the same pattern. On the 1.5B model the standard deviation of the final Average Format over five seeds is 0.34 for DARA-Asym and 0.64 for DARA-Sym. It is 1.00 for GDPO and 1.36 for GD$^2$PO-Hard. It reaches 2.91 for DVAO and 5.18 for GRPO.