- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies sample reuse in RLVR. Reusing rollout batches is attractive because rollout generation is expensive, but repeated optimization on the same batch can amplify policy shift and cause training collapse.
The authors identify a phenomenon called Disproportionate Weight Divergence (DWD): at the onset of performance degradation, the weight change of the increases sharply while intermediate attention and MLP layers remain relatively stable. This pattern is observed across multiple Qwen and Llama models and across math, ALFWorld, WebShop, and search-augmented QA.
The paper provides two theoretical explanations:
- the directly receives the output error signal, whereas gradients to intermediate layers are attenuated through downstream Jacobian projections;
- the gradient norm is structurally related to empirical Pearson policy divergence.
Based on these observations, the authors propose Dynamic Gradient Gating (DGG). DGG monitors the increment of the gradient energy with a Z-score test. Once an anomalous spike is detected, the gradient is discarded before entering Adam and reuse of the current batch is terminated.
Across reasoning and agentic tasks, DGG achieves approximately 2–3× rollout efficiency and up to 2.14× wall-clock speedup.
Critical strengths
-
The problem is important and practically relevant.
Rollout cost is a major bottleneck in RLVR, so determining how long a batch can be safely reused is an important systems and optimization problem.
-
DWD is a memorable and empirically clear phenomenon.
The localized divergence appears strongly synchronized with training collapse, while the intermediate layers remain comparatively stable.
-
The empirical-discovery → theory → algorithm pipeline is coherent.
The method is motivated by a concrete observed failure signal rather than by an arbitrary heuristic.
-
The monitoring-signal comparison is convincing.
KL divergence, clip ratio, and global gradient norm do not provide the same sharp collapse signal as the gradient norm.
-
Adaptive reuse clearly improves over fixed reuse.
Fixed reuse exhibits the expected efficiency–stability trade-off, whereas DGG dynamically selects reuse depth according to the observed gradient signal.
-
The validation is broad for an RLVR paper.
The method is tested across models, tasks, and optimization algorithms, with additional appendix results extending to Llama, DAPO, and 14B models.
Critical weaknesses
- Theorem 2 can be easily overinterpreted.$
$
The actual statement is
where $�1�$ depends on the advantage, representation, and output-distribution scale.
Thus, the gradient norm is not a calibrated divergence estimator; it is a structurally motivated signal.
2. DWD is established as a strong indicator, but not necessarily as the causal driver of collapse.
The success of DGG shows that the signal is useful, but does not by itself prove that $�2�$ divergence causes the collapse.
3. The Z-score detector is heuristic.
The novelty lies more in the signal than in the anomaly detector. It would be useful to establish whether the threshold and running-statistics design are robust across larger scales and more training regimes.
4. Gradient spikes can in principle be beneficial.
A large update may correspond to a useful behavior transition rather than a harmful shift. It is unclear whether DGG could suppress such updates in settings where large policy jumps are desirable.
5. The theory does not yet cover full actor–critic PPO.
The paper explicitly notes that the analysis only covers the clipped policy-gradient component and not gradients from value-function optimization through a shared backbone.
6. The main value is efficiency rather than a large final-performance gain.
This is not a problem per se, but it makes accurate wall-clock accounting and rollout-cost control particularly important.
Questions for authors
- Can the authors selectively clip or freeze gradients to test whether DWD is causal rather than merely correlated with collapse?
- How stable is during training, and how does its variation affect the interpretation of raw gradient norm?
- Why is the increment preferable to alternatives such as a percentile threshold, EWMA ratio, or normalized gradient magnitude?
2 comments
Response to Reviewer xQ1E [Part II]
W4: Potential Suppression of Beneficial Gradient Updates.
RW4: Thank you for highlighting this consideration. We agree that large gradient updates may be beneficial. DGG is designed to mitigate this risk by (1) detecting anomalous changes relative to recent gradient statistics, rather than simply suppressing large gradients, and (2) activating gating only during sample reuse, leaving the first update on each fresh rollout batch unaffected.
Our experimental results across diverse models and tasks further suggest that the benefits of preventing harmful updates outweigh the potential cost of discarding beneficial ones, as evidenced by improved sample efficiency without sacrificing final performance.
W5: Applicability to Actor–Critic PPO.
RW5: Thank you for highlighting this important distinction. We would like to clarify that our theoretical analysis applies to the clipped policy-gradient component of PPO, addressing reuse-induced policy drift and gradient amplification during actor updates. As already acknowledged in Section 7 (Conclusion and Limitations), empirical evaluation under full actor–critic PPO is beyond our scope and remains future work. We have further revised this section to clarify the theoretical scope.
W6: Questions About Efficiency Gains
RW6: Thank you for highlighting this point. Sample reuse aims to improve efficiency but often suffers from severe training instability. DGG’s primary contribution is to address this instability, enabling effective reuse without sacrificing final performance.
We agree that accurate efficiency evaluation is essential. Our original manuscript already reports both rollout and wall-clock speedups in Table 1, with DGG achieving up to 2.93$\times$ and 2.14$\times$, respectively, over the single-use baseline. Furthermore, Appendix G.2 already provides a detailed runtime breakdown, accounting for both the reduction in rollout generation cost and the additional policy-update overhead introduced by sample reuse.
Q3: Compatibility with Alternative Anomaly Detectors
RQ3: Thank you for your constructive suggestion. Following your suggestion, we conducted additional experiments by replacing the original Z-score detector with an EWMA-based alternative while retaining the same gradient monitoring signal. As shown in Table R7, DGG with the EWMA detector consistently outperforms the single-use baseline across all four benchmarks, achieving sample efficiency speedups ranging from to . These results suggest that DGG's effectiveness is not tied to the Z-score detector, further supporting the usefulness of the proposed lm_head gradient signal for mitigating training instability during sample reuse.
Table R7: Accuracy (%) of GRPO without reuse and DGG with an EWMA-based detector on Qwen2.5-7B-Instruct across four benchmarks at different RL steps. The best result at each checkpoint is highlighted in bold.
| Benchmark | Method | Step 50 | Step 100 | Step 150 | Step 200 | Step 250 | Step 300 | Step 350 | Step 400 | Step 450 | Step 500 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Math500 | GRPO w/o Reuse | 75.7% | 77.1% | 77.5% | 77.6% | 77.4% | 77.6% | 77.5% | 77.2% | 77.3% | 77.5% |
| GRPO w/ DGG (EWMA Detector) | 76.9% | 77.6% | 77.7% | 77.9% | 78.2% | 78.3% | 78.5% | 78.7% | 78.7% | 78.7% | |
| Minerva | GRPO w/o Reuse | 28.0% | 28.2% | 28.4% | 28.2% | 28.3% | 28.3% | 28.5% | 28.2% | 28.5% | 28.5% |
| GRPO w/ DGG (EWMA Detector) | 28.1% | 28.5% | 28.6% | 28.6% | 28.8% | 28.9% | 29.1% | 28.9% | 29.2% | 29.1% | |
| AIME25 | GRPO w/o Reuse | 11.2% | 13.4% | 12.4% | 14.3% | 14.0% | 14.5% | 14.4% | 16.1% | 14.1% | 16.2% |
| GRPO w/ DGG (EWMA Detector) | 12.6% | 13.9% | 14.6% | 16.3% | 15.6% | 16.5% | 16.7% | 16.5% | 16.7% | 18.1% | |
| Olympiad | GRPO w/o Reuse | 36.3% | 37.9% | 37.9% | 38.2% | 37.9% | 38.1% | 38.4% | 38.1% | 38.5% | 38.4% |
| GRPO w/ DGG (EWMA Detector) | 37.3% | 38.0% | 38.4% | 39.1% | 39.3% | 39.0% | 38.9% | 39.1% | 38.9% | 39.4% |
In the revised manuscript, we have added a discussion of compatibility with alternative anomaly detectors in Section 6.3 and included the complete training curve comparisons in Appendix L (Compatibility with Alternative Anomaly Detectors).
Response to Reviewer xQ1E [Part I]
We sincerely thank the reviewer for recognizing the practical importance of the problem, the clear empirical evidence for DWD, the coherent progression from discovery to theory and algorithm design, the effectiveness of our monitoring signal and adaptive reuse, and our broad validation across models, tasks, and optimization algorithms. We respond to each concern point by point below. All new results and clarifications are included in the revised manuscript and highlighted in blue.
RW1 & RQ2: Thank you for your insightful comment. We agree that the lm_head gradient norm is a structurally motivated monitoring signal rather than a calibrated estimator of policy divergence. As explicitly stated in Section 4.2.2 of our original manuscript:
Therefore, the potential variation of cmax during training does not invalidate Theorem 2, whose role is to provide a structural motivation for gradient-based monitoring.
RW2 & RQ1: Thank you for your constructive comments. We agree that DWD is established as a strong indicator of training collapse, rather than its causal driver. Its consistent correlation with performance degradation motivates DGG’s detection of anomalous gradients to prevent harmful updates.
Following your suggestion, we froze the lm_head throughout training. Table R5 shows that accuracy initially improves but subsequently declines, indicating that suppressing DWD through lm_head freezing is insufficient to prevent training collapse and providing further evidence that DWD is not the fundamental cause of training instability.
Table R5: Accuracy (%) with the lm_head frozen on Qwen2.5-7B-Instruct (Math500) at different RL steps.
Regarding the fundamental cause of training instability, we have already analyzed the underlying mechanism in Section 3 (The Vulnerability of Sample Reuse in LLMs): repeated updates increase the mismatch between the updating and behavior policies, inflating importance ratios for low-probability tokens and amplifying their gradients, particularly for unclipped negative-advantage tokens, ultimately destabilizing training.
RW3: Thank you for your insightful comment. We agree that our primary contribution is identifying the lm_head gradient norm as an effective monitoring signal, rather than introducing a novel detector. Its effectiveness with a simple Z-score test demonstrates its practical utility.
Regarding robustness, we would like to clarify that all main experiments use the same configuration, with W=20 and τ=0.5, across all models and tasks, without model-specific or task-specific tuning. Consistent improvements across tasks, model scales (1.5B–14B), and algorithms (GRPO and DAPO) demonstrate robustness under this shared configuration.
Section 6.3 already provides a sensitivity analysis of the anomaly threshold τ, showing improvements across different settings. We additionally evaluate W∈{10,20,30} on Qwen3-4B-Instruct (Math500). Table R6 shows that DGG outperforms the single-use baseline across all evaluated window sizes, supporting robustness to both τ and W.
Table R6: Accuracy (%) of DGG with different sliding window sizes W on Qwen3-4B-Instruct (Math500) at different RL steps.
In the revised manuscript, we have added a sensitivity analysis of the sliding window size W in Appendix K (Hyperparameter Sensitivity Analysis), with a corresponding reference added to Section 6.3 (Sensitivity Analysis and Practical Recommendations).