acceptodds
Under review as a conference paper at ICLR 2027

When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs), but rollout samples are expensive to obtain, making sample efficiency a critical bottleneck. A natural remedy is to reuse each rollout batch for multiple gradient updates, a standard practice in classical RL. Yet in RLVR, this amplifies policy shift, leading to severe performance degradation. Detecting the onset of degradation early enough to stop reuse remains an open and challenging problem. We close this gap by identifying the Disproportionate Weight Divergence (DWD) phenomenon: performance degradation is synchronized with a sharp surge in the lm_head weight change, while intermediate layers remain stable. Empirically, we verify that DWD emerges consistently across diverse LLMs and tasks. Theoretically, we prove that (i) harmful gradients concentrate at the lm_head while intermediate layers are structurally attenuated, and (ii) the lm_head gradient norm lower-bounds the policy divergence. These results establish the lm_head gradient norm as a principled, real-time signal of catastrophic policy shift. Guided by this insight, we propose Dynamic Gradient Gating (DGG), a lightweight intervention that monitors the lm_head gradient norm in real time and intercepts harmful gradients before they corrupt the optimizer. DGG consistently matches or exceeds the standard single-use baseline, achieving up to sample efficiency and wall-clock speedup across math, ALFWorld, WebShop, and search-augmented QA tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, LLM, RLVR, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xQ1EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sample reuse in RLVR. Reusing rollout batches is attractive because rollout generation is expensive, but repeated optimization on the same batch can amplify policy shift and cause training collapse.

The authors identify a phenomenon called Disproportionate Weight Divergence (DWD): at the onset of performance degradation, the weight change of the lmheadlm_head increases sharply while intermediate attention and MLP layers remain relatively stable. This pattern is observed across multiple Qwen and Llama models and across math, ALFWorld, WebShop, and search-augmented QA.

The paper provides two theoretical explanations:

  • the �1��1� directly receives the output error signal, whereas gradients to intermediate layers are attenuated through downstream Jacobian projections;
  • the �2��2� gradient norm is structurally related to empirical Pearson �3��3� policy divergence.

Based on these observations, the authors propose Dynamic Gradient Gating (DGG). DGG monitors the increment of the �4��4� gradient energy with a Z-score test. Once an anomalous spike is detected, the gradient is discarded before entering Adam and reuse of the current batch is terminated.

Across reasoning and agentic tasks, DGG achieves approximately 2–3× rollout efficiency and up to 2.14× wall-clock speedup.

Critical strengths
  1. The problem is important and practically relevant.

    Rollout cost is a major bottleneck in RLVR, so determining how long a batch can be safely reused is an important systems and optimization problem.

  2. DWD is a memorable and empirically clear phenomenon.

    The localized lmheadlm_head divergence appears strongly synchronized with training collapse, while the intermediate layers remain comparatively stable.

  3. The empirical-discovery → theory → algorithm pipeline is coherent.

    The method is motivated by a concrete observed failure signal rather than by an arbitrary heuristic.

  4. The monitoring-signal comparison is convincing.

    KL divergence, clip ratio, and global gradient norm do not provide the same sharp collapse signal as the lmheadlm_head gradient norm.

  5. Adaptive reuse clearly improves over fixed reuse.

    Fixed reuse exhibits the expected efficiency–stability trade-off, whereas DGG dynamically selects reuse depth according to the observed gradient signal.

  6. The validation is broad for an RLVR paper.

    The method is tested across models, tasks, and optimization algorithms, with additional appendix results extending to Llama, DAPO, and 14B models.

Critical weaknesses
  1. Theorem 2 can be easily overinterpreted.$
χ2≥∥Glm∥2cmax⁡−1,\chi^2 \ge \frac{\|G^{lm}\|^2}{c_{\max}} - 1,

$

The actual statement is

where $�1�$ depends on the advantage, representation, and output-distribution scale.    

Thus, the gradient norm is not a calibrated divergence estimator; it is a structurally motivated signal.

2. DWD is established as a strong indicator, but not necessarily as the causal driver of collapse.

The success of DGG shows that the signal is useful, but does not by itself prove that $�2�$ divergence causes the collapse.

3. The Z-score detector is heuristic.

The novelty lies more in the signal than in the anomaly detector. It would be useful to establish whether the threshold and running-statistics design are robust across larger scales and more training regimes.

4. Gradient spikes can in principle be beneficial.

A large update may correspond to a useful behavior transition rather than a harmful shift. It is unclear whether DGG could suppress such updates in settings where large policy jumps are desirable.

5. The theory does not yet cover full actor–critic PPO.

The paper explicitly notes that the analysis only covers the clipped policy-gradient component and not gradients from value-function optimization through a shared backbone.   

6. The main value is efficiency rather than a large final-performance gain.

This is not a problem per se, but it makes accurate wall-clock accounting and rollout-cost control particularly important.
Questions for authors
  1. Can the authors selectively clip or freeze �0��0� gradients to test whether DWD is causal rather than merely correlated with collapse?
  2. How stable is �1��1� during training, and how does its variation affect the interpretation of raw �2��2� gradient norm?
  3. Why is the increment �3��3� preferable to alternatives such as a percentile threshold, EWMA ratio, or normalized gradient magnitude?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xQ1E [Part I]

We sincerely thank the reviewer for recognizing the practical importance of the problem, the clear empirical evidence for DWD, the coherent progression from discovery to theory and algorithm design, the effectiveness of our monitoring signal and adaptive reuse, and our broad validation across models, tasks, and optimization algorithms. We respond to each concern point by point below. All new results and clarifications are included in the revised manuscript and highlighted in blue.


W1 & Q2: Theoretical Interpretation of Theorem 2.

RW1 & RQ2: Thank you for your insightful comment. We agree that the lm_head gradient norm is a structurally motivated monitoring signal rather than a calibrated estimator of policy divergence. As explicitly stated in Section 4.2.2 of our original manuscript:

“Theorem 2 thus establishes a structural link between the lm_head gradient energy and off-policy policy shift, motivating the lm_head gradient norm as the monitoring signal.”

Therefore, the potential variation of cmax⁡c_{\max} during training does not invalidate Theorem 2, whose role is to provide a structural motivation for gradient-based monitoring.


W2 & Q1: Causal Role of DWD in Training Collapse.

RW2 & RQ1: Thank you for your constructive comments. We agree that DWD is established as a strong indicator of training collapse, rather than its causal driver. Its consistent correlation with performance degradation motivates DGG’s detection of anomalous gradients to prevent harmful updates.

Following your suggestion, we froze the lm_head throughout training. Table R5 shows that accuracy initially improves but subsequently declines, indicating that suppressing DWD through lm_head freezing is insufficient to prevent training collapse and providing further evidence that DWD is not the fundamental cause of training instability.

Table R5: Accuracy (%) with the lm_head frozen on Qwen2.5-7B-Instruct (Math500) at different RL steps.

MethodStep 0Step 50Step 100Step 150Step 200
GRPO w/ Naive Reuse and Frozen lm_head74.977.478.176.473.3

Regarding the fundamental cause of training instability, we have already analyzed the underlying mechanism in Section 3 (The Vulnerability of Sample Reuse in LLMs): repeated updates increase the mismatch between the updating and behavior policies, inflating importance ratios for low-probability tokens and amplifying their gradients, particularly for unclipped negative-advantage tokens, ultimately destabilizing training.


W3: Z-score Detector Design and Robustness.

RW3: Thank you for your insightful comment. We agree that our primary contribution is identifying the lm_head gradient norm as an effective monitoring signal, rather than introducing a novel detector. Its effectiveness with a simple Z-score test demonstrates its practical utility.

Regarding robustness, we would like to clarify that all main experiments use the same configuration, with W=20W=20 and τ=0.5\tau=0.5, across all models and tasks, without model-specific or task-specific tuning. Consistent improvements across tasks, model scales (1.5B–14B), and algorithms (GRPO and DAPO) demonstrate robustness under this shared configuration.

Section 6.3 already provides a sensitivity analysis of the anomaly threshold τ\tau, showing improvements across different settings. We additionally evaluate W∈{10,20,30}W \in \{10, 20, 30\} on Qwen3-4B-Instruct (Math500). Table R6 shows that DGG outperforms the single-use baseline across all evaluated window sizes, supporting robustness to both τ\tau and WW.

Table R6: Accuracy (%) of DGG with different sliding window sizes WW on Qwen3-4B-Instruct (Math500) at different RL steps.

MethodStep 50Step 100Step 150Step 200Step 250Step 300Step 350Step 400Step 450Step 500
GRPO w/o Reuse88.789.689.689.990.490.490.690.590.590.3
DGG (W=10)89.390.289.991.091.191.591.991.391.792.1
DGG (W=20)89.489.990.591.091.791.791.591.591.991.7
DGG (W=30)89.089.890.491.291.191.191.591.191.791.9

In the revised manuscript, we have added a sensitivity analysis of the sliding window size WW in Appendix K (Hyperparameter Sensitivity Analysis), with a corresponding reference added to Section 6.3 (Sensitivity Analysis and Practical Recommendations).

AuthorsOpenReview ↗

Response to Reviewer xQ1E [Part II]

W4: Potential Suppression of Beneficial Gradient Updates.

RW4: Thank you for highlighting this consideration. We agree that large gradient updates may be beneficial. DGG is designed to mitigate this risk by (1) detecting anomalous changes relative to recent gradient statistics, rather than simply suppressing large gradients, and (2) activating gating only during sample reuse, leaving the first update on each fresh rollout batch unaffected.

Our experimental results across diverse models and tasks further suggest that the benefits of preventing harmful updates outweigh the potential cost of discarding beneficial ones, as evidenced by improved sample efficiency without sacrificing final performance.


W5: Applicability to Actor–Critic PPO.

RW5: Thank you for highlighting this important distinction. We would like to clarify that our theoretical analysis applies to the clipped policy-gradient component of PPO, addressing reuse-induced policy drift and gradient amplification during actor updates. As already acknowledged in Section 7 (Conclusion and Limitations), empirical evaluation under full actor–critic PPO is beyond our scope and remains future work. We have further revised this section to clarify the theoretical scope.


W6: Questions About Efficiency Gains

RW6: Thank you for highlighting this point. Sample reuse aims to improve efficiency but often suffers from severe training instability. DGG’s primary contribution is to address this instability, enabling effective reuse without sacrificing final performance.

We agree that accurate efficiency evaluation is essential. Our original manuscript already reports both rollout and wall-clock speedups in Table 1, with DGG achieving up to 2.93$\times$ and 2.14$\times$, respectively, over the single-use baseline. Furthermore, Appendix G.2 already provides a detailed runtime breakdown, accounting for both the reduction in rollout generation cost and the additional policy-update overhead introduced by sample reuse.


Q3: Compatibility with Alternative Anomaly Detectors

RQ3: Thank you for your constructive suggestion. Following your suggestion, we conducted additional experiments by replacing the original Z-score detector with an EWMA-based alternative while retaining the same gradient monitoring signal. As shown in Table R7, DGG with the EWMA detector consistently outperforms the single-use baseline across all four benchmarks, achieving sample efficiency speedups ranging from 2.17×2.17\times to 3.08×3.08\times. These results suggest that DGG's effectiveness is not tied to the Z-score detector, further supporting the usefulness of the proposed lm_head gradient signal for mitigating training instability during sample reuse.

Table R7: Accuracy (%) of GRPO without reuse and DGG with an EWMA-based detector on Qwen2.5-7B-Instruct across four benchmarks at different RL steps. The best result at each checkpoint is highlighted in bold.

BenchmarkMethodStep 50Step 100Step 150Step 200Step 250Step 300Step 350Step 400Step 450Step 500
Math500GRPO w/o Reuse75.7%77.1%77.5%77.6%77.4%77.6%77.5%77.2%77.3%77.5%
GRPO w/ DGG (EWMA Detector)76.9%77.6%77.7%77.9%78.2%78.3%78.5%78.7%78.7%78.7%
MinervaGRPO w/o Reuse28.0%28.2%28.4%28.2%28.3%28.3%28.5%28.2%28.5%28.5%
GRPO w/ DGG (EWMA Detector)28.1%28.5%28.6%28.6%28.8%28.9%29.1%28.9%29.2%29.1%
AIME25GRPO w/o Reuse11.2%13.4%12.4%14.3%14.0%14.5%14.4%16.1%14.1%16.2%
GRPO w/ DGG (EWMA Detector)12.6%13.9%14.6%16.3%15.6%16.5%16.7%16.5%16.7%18.1%
OlympiadGRPO w/o Reuse36.3%37.9%37.9%38.2%37.9%38.1%38.4%38.1%38.5%38.4%
GRPO w/ DGG (EWMA Detector)37.3%38.0%38.4%39.1%39.3%39.0%38.9%39.1%38.9%39.4%

In the revised manuscript, we have added a discussion of compatibility with alternative anomaly detectors in Section 6.3 and included the complete training curve comparisons in Appendix L (Compatibility with Alternative Anomaly Detectors).

Reviewer_hZSJOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyze the problem of reusing rollout, and it proposes DGG, which use gradient norms in lm_head to drop some broken gradient to make RLVR more stable.

Critical strengths
  1. A full paths from obseverions, analyze, method and verification.
  2. Many experiments run on many models with different sizes.
  3. All methods use the same hyperparameters in each task and share the same rollout budget at each step. The paper also runs 3 seeds, shows sensitivity to τ and K, and releases the code.
Critical weaknesses
  1. Lack comparison with many rollout replay methods. They save rollouts for the same goal.
Questions for authors
  1. Does it scaling? I know it is expensive, but it is better to show the comparison from 1B to 14B.
  2. How is τ selected? The paper only says it is chosen from {0.1, 0.5, 1.0}, but not on which set or by which rule. The baselines do not have such a knob.
1 comment
AuthorsOpenReview ↗

Response to Reviewer hZSJ

We sincerely thank the reviewer for recognizing the coherent progression from observation and analysis to method design and validation, our extensive experiments across models and scales, and the fairness and reproducibility of our evaluation. We respond to each concern point by point below. All new results and clarifications are included in the revised manuscript and highlighted in blue.


W1: Comparison with Rollout Replay Methods.

RW1: Thank you for your constructive suggestion. DGG is orthogonal to rollout replay methods, as discussed in Section 2 (Related Work). We further evaluate their combination by integrating DGG with a representative rollout replay method [1]. As shown in Table R3, adding DGG consistently improves performance on both Math500 and Olympiad compared with rollout replay alone under the same rollout generation budget, demonstrating their compatibility and the additional gains provided by DGG.

Table R3: Accuracy (%) of GRPO with Rollout Replay (RR), with and without DGG, on Qwen3-4B-Instruct at different cumulative rollout batches. The best result at each checkpoint is highlighted in bold.

BenchmarkMethodBatch 50Batch 100Batch 150Batch 200Batch 250Batch 300Batch 350Batch 400Batch 450Batch 500
Math500GRPO w/ RR89.089.390.290.990.690.891.191.391.390.9
GRPO w/ RR and DGG (Ours)89.590.391.391.591.791.691.792.092.192.0
OlympiadGRPO w/ RR54.255.556.457.157.257.658.157.958.157.5
GRPO w/ RR and DGG (Ours)55.156.757.858.158.058.258.158.558.258.4

In the revised manuscript, we have incorporated the complete training curve comparisons of rollout replay with and without DGG into Section 6.3 (Figure 7).

[1] Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay, NeurIPS 2026.


Q1: Does DGG Scale from 1B to 14B?

RQ1: Thank you for your valuable suggestion. To further investigate the scalability of DGG, we conduct additional experiments on Qwen2.5-1.5B-Instruct, extending our existing evaluation on 7B and 14B models to cover three model scales ranging from 1.5B to 14B under DAPO. As shown in Table R4, DGG consistently improves sample efficiency across all three model scales and four benchmarks, demonstrating its effectiveness across a wide range of model scales.

Table R4: Sample efficiency improvement of DGG under DAPO over the single-use baseline across different model scales and benchmarks. The AMC23/AIME25 column reports results on AMC23 for the 1.5B model and AIME25 for the 7B and 14B models.

ModelMath500AMC23 / AIME25MinervaOlympiad
Qwen2.5-1.5B-Instruct1.83×2.14×3.45×2.29×
Qwen2.5-7B-Instruct1.83×2.20×3.43×2.58×
Qwen2.5-14B-Instruct2.33×2.67×3.40×1.86×

In the revised manuscript, we have added a cross-scale sample efficiency comparison in Section 6.3 (Table 2) and extended the complete training curve comparisons in Appendix E.2 (Figure 12) to cover Qwen2.5 models ranging from 1.5B to 14B parameters under DAPO.


Q2: Questions About τ\tau Selection.

RQ2: Thank you for your constructive suggestion. We would like to clarify that all main experiments use the same anomaly threshold τ=0.5\tau=0.5 across all models and tasks, without model-specific, task-specific, or benchmark-specific tuning. The consistent improvements across model families, scales, and tasks support the effectiveness of this shared threshold.

Additionally, the candidate set τ∈{0.1,0.5,1.0}\tau \in \{0.1, 0.5, 1.0\} recommended in the original manuscript provides practical adjustment guidance, rather than describing threshold selection for the main experiments. Section 6.3 already provides explicit guidance on threshold selection, recommending smaller values to prioritize training stability, given the asymmetric cost of mis-tuning: conservative thresholds sacrifice some reuse, whereas overly large thresholds risk performance degradation.

We have revised Section 6.1 (Training Details) to explicitly state the fixed threshold used in our main experiments and clarified the purpose of the recommended threshold range in Section 6.3 (Sensitivity Analysis and Practical Recommendations).

Reviewer_ovGqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training degradation from rollout reuse in RLVR and observes that performance drops often coincide with spikes in the relative weight change of lm_head. It proposes DGG, which monitors increases in lm_head gradient energy. When an anomaly is detected, DGG discards the gradients before the Adam step and stops reusing the current batch. Experiments on reasoning and interactive tasks show better sample utilization and training time efficiency while maintaining performance.

Critical strengths
  1. An important problem with a concrete observation. The paper addresses the effective reuse of costly rollouts and shows DWD across multiple Qwen and Llama configurations and tasks, offering a useful perspective on diagnosing instability during reuse.
  2. Broad experimental coverage. Comparisons cover different fixed reuse counts, multiple models, and multiple tasks, with some experiments run over three seeds. The reported training efficiency gains also demonstrate the method’s practical value.
Critical weaknesses
  1. The benefit of skipping the flagged update is not isolated. DGG both skips the current flagged update and stops further reuse. The experiments support the combined mechanism, but do not show how much of the gain comes from avoiding the current update versus reducing subsequent off-policy updates.

  2. More hyperparameter sensitivity analysis is needed. The implementation fixes (W=20) and selects τ\tau from ({0.1, 0.5, 1.0}), but does not clearly report the threshold for each configuration or how it was chosen. The sensitivity analysis in Figure 5 focuses mainly on Qwen3-4B/Math500, so it remains unclear whether a single default setting works reliably across tasks.

Questions for authors

What thresholds are used for each model and task, and what data and criteria were used to select them? Could you report results with a single default threshold on representative tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ovGq [Part I]

We sincerely thank the reviewer for recognizing the importance of the problem, the diagnostic value of DWD, and our broad experimental coverage and practical efficiency gains. We respond to each concern point by point below. All new results and clarifications are included in the revised manuscript and highlighted in blue.


W1: Contribution of the Gradient Discard Mechanism.

RW1: Thank you for your insightful comment. To address it, we conduct an additional ablation study by introducing DGG w/o Discard, which uses the same anomaly detection and reuse termination mechanism as full DGG, but executes the flagged optimizer update instead of discarding it. As shown in Table R1, DGG w/o Discard exhibits substantial performance degradation in later training stages, whereas full DGG maintains considerably more stable performance. These results demonstrate that reuse termination alone is insufficient to prevent training degradation, highlighting the critical role of gradient discard in maintaining training stability.

Table R1: Accuracy (%) of DGG with and without gradient discard on Qwen3-4B-Instruct across four benchmarks at different RL steps. The best result at each checkpoint is highlighted in bold.

BenchmarkMethodStep 50Step 100Step 150Step 200Step 250Step 300Step 350Step 400Step 450Step 500
Math500DGG w/o Discard89.3%90.1%90.2%91.3%90.9%91.3%91.0%90.8%90.3%89.8%
DGG (Full)89.4%89.9%90.5%91.0%91.7%91.7%91.5%91.5%91.9%91.7%
MinervaDGG w/o Discard33.2%33.8%34.8%33.8%34.1%34.6%34.2%33.9%34.5%33.9%
DGG (Full)33.7%34.2%34.8%34.7%34.6%34.7%34.7%35.1%35.4%35.3%
AIME25DGG w/o Discard32.8%32.0%34.0%37.4%35.9%35.1%31.3%32.5%31.3%33.9%
DGG (Full)33.3%32.8%37.9%38.5%37.3%39.1%40.0%41.0%39.2%38.6%
OlympiadDGG w/o Discard53.9%55.8%56.2%56.6%57.5%56.2%55.6%55.9%55.7%55.0%
DGG (Full)54.1%56.2%57.2%57.0%57.5%57.9%57.8%58.2%58.4%57.9%

We would like to clarify that gradient discard and reuse termination are inherently coupled in DGG. Once an anomalous gradient is detected, it signals a substantial risk of harmful policy updates, motivating both the rejection of the current update and the immediate termination of further reuse to avoid further increasing off-policy divergence.

In the revised manuscript, we have incorporated this ablation study into Section 6.3 (Figure 6), with complete training curve comparisons across all four benchmarks provided in Appendix H (Figure 14).

AuthorsOpenReview ↗

Response to Reviewer ovGq [Part II]

W2 & Q1: Questions About Hyperparameter Selection.

RW2 & RQ1: Thank you for your constructive suggestion. All main experiments use the same configuration, with W=20W=20 and τ=0.5\tau=0.5, across all models and tasks, without task-specific or model-specific tuning. Consistent improvements across model families, scales, and tasks demonstrate the effectiveness of this shared configuration.

Additionally, the candidate set τ∈{0.1,0.5,1.0}\tau \in \{0.1, 0.5, 1.0\} recommended in the original manuscript provides practical adjustment guidance, rather than describing threshold selection for the main experiments. Section 6.3 already provides explicit guidance on threshold selection, recommending smaller values to prioritize training stability, given the asymmetric cost of mis-tuning: conservative thresholds sacrifice some reuse, whereas overly large thresholds risk performance degradation.

To complement the existing sensitivity analysis of τ\tau in Section 6.3, we additionally evaluate W∈{10,20,30}W \in \{10,20,30\} on Qwen3-4B-Instruct (Math500). As shown in Table R2, DGG consistently outperforms the single-use baseline across all evaluated window sizes, further supporting robustness to both τ\tau and WW.

Table R2: Accuracy (%) of DGG with different sliding window sizes WW on Qwen3-4B-Instruct (Math500) at different RL steps.

MethodStep 50Step 100Step 150Step 200Step 250Step 300Step 350Step 400Step 450Step 500
GRPO w/o Reuse88.789.689.689.990.490.490.690.590.590.3
DGG (W=10)89.390.289.991.091.191.591.991.391.792.1
DGG (W=20)89.489.990.591.091.791.791.591.591.991.7
DGG (W=30)89.089.890.491.291.191.191.591.191.791.9

We have clarified the fixed hyperparameter configuration in Section 6.1 (Training Details) and the purpose of the recommended threshold range in Section 6.3 (Sensitivity Analysis and Practical Recommendations), and added the WW sensitivity analysis in Appendix K (Hyperparameter Sensitivity Analysis).

Comments on the paper

2 comments

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. If the provided responses do not address (even in part) your concerns, this is your chance to raise them. Please engage with the authors promptly. We really appreciate your efforts! Thanks, AC

AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We sincerely thank all reviewers for their constructive feedback.

Our work addresses training instability caused by sample reuse in RLVR. We identify the Disproportionate Weight Divergence (DWD) phenomenon, where performance degradation coincides with a sharp surge in lm_head weight changes while intermediate layers remain stable. We theoretically explain this structural asymmetry and establish a lower-bound relationship between the lm_head gradient norm and policy divergence. Building on these insights, we propose Dynamic Gradient Gating (DGG) to intercept harmful gradients and dynamically terminate sample reuse, achieving up to 2.93× sample efficiency and 2.14× wall-clock speedup.

Below, we summarize the reviewers' positive assessments of our work and the major improvements made in response to their feedback.

1. Reviewers' Recognition of the Work

We appreciate the reviewers' recognition of several key aspects of our work:

  • Importance and practical relevance (ovGq, xQ1E). The reviewers recognized the importance of improving rollout utilization in RLVR and the practical value of addressing training instability caused by sample reuse.
  • Empirical discovery and diagnostic value of DWD (ovGq, xQ1E). The reviewers highlighted DWD as a meaningful and empirically clear observation, providing a useful signal for identifying training instability during sample reuse.
  • Coherent progression from observation to theory and algorithm (hZSJ, xQ1E). The reviewers appreciated the connection between empirical findings, theoretical analysis, and the design of DGG. Reviewer xQ1E also recognized the effectiveness of the proposed monitoring signal compared with conventional alternatives.
  • Broad validation and practical efficiency gains (ovGq, hZSJ, xQ1E). The reviewers acknowledged the extensive evaluation across different models, tasks, and optimization algorithms, as well as the demonstrated improvements in sample efficiency and wall-clock training efficiency.

2. Major Improvements During Rebuttal

In response to the reviewers' feedback, we conducted additional experiments and revised the manuscript, with all changes highlighted in blue, as summarized below.

  1. Contribution of Gradient Discard (ovGq: W1). We added an ablation study demonstrating that reuse termination alone is insufficient to prevent training degradation, highlighting the critical role of gradient discard in maintaining training stability. See the revisions in Section 6.3 (Pages 9–10, Figure 6) and Appendix H (Pages 29–30, Figure 14).
  2. Hyperparameter Selection and Robustness (ovGq: W2, Q1; hZSJ: Q2; xQ1E: W3). We clarified that all main experiments use the same configuration (W=20W=20, τ=0.5\tau=0.5) without model-specific or task-specific tuning. Additional sensitivity analysis of the sliding window size WW further demonstrate the robustness of DGG to hyperparameter choices. See the revisions in Section 6.1 (Page 7), Section 6.3 (Page 9), and Appendix K, Figure 16 (Page 32).
  3. Compatibility with Rollout Replay (hZSJ: W1). We combined DGG with a representative rollout replay method, demonstrating their compatibility and the additional performance gains provided by DGG. See the revisions in Section 6.3, Figure 7 (Page 10).
  4. Scalability Across Model Sizes (hZSJ: Q1). We extended DAPO experiments to models ranging from 1.5B to 14B, further demonstrating the effectiveness of DGG across different model scales. See the revisions in Section 6.3, Table 2 (Page 10), and Appendix E.2, Figure 12 (Page 27).
  5. Theoretical Interpretation and the Role of DWD (xQ1E: W1, W2, Q1, Q2). We clarified the theoretical connection between the lm_head gradient norm and policy divergence established by Theorem 2. Additional frozen-lm_head experiments further support the role of DWD as a monitoring indicator of training instability. See the clarifications in Section 4.2.2 (Page 6) and our response to Reviewer xQ1E.
  6. Robustness to Anomaly Detection Mechanisms (xQ1E: W3, Q3). We introduced an EWMA-based alternative to the Z-score detector. The consistent effectiveness of both detectors further supports the central contribution of our work: identifying an effective and robust lm_head gradient monitoring signal for mitigating training instability during sample reuse. See the revisions in Section 6.3 (Page 10) and Appendix L, Figure 17 (Page 32).

We also provided clarifications on additional points regarding potentially beneficial gradient updates (xQ1E: W4), the scope of actor–critic PPO analysis (xQ1E: W5), and runtime efficiency accounting (xQ1E: W6) in our rebuttal responses.

Overall, we hope that our additional experiments and clarifications have addressed the reviewers' concerns. These efforts have further strengthened the empirical evidence and practical value of DGG for stable and sample-efficient RLVR training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.