acceptodds
Desk rejected at ICLR 2027

Scaling Laws for Collapse in Asynchronous GRPO

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Asynchronous reinforcement learning improves the throughput of large language model post-training by decoupling rollout generation from policy optimization, but introduces a mismatch between the behavior and learner policies. How the resulting policy staleness couples with the learning rate to govern training stability and collapse time remains poorly understood. We investigate this coupling in vanilla GRPO through controlled sweeps of the synchronization interval S and constant learning rate η on Llama-3.2-1B/3B, complemented by experiments on Qwen3-8B. We identify two empirical scaling laws: (i) Stability-boundary scaling: the largest stable learning rate scales approximately as S⁻¹, yielding a stability boundary characterized by an approximately constant product Sη. (ii) Collapse-time scaling: among collapsing runs, estimated collapse times scale approximately as η⁻¹, corresponding to a model- and setup-dependent cumulative learning-rate budget that aligns across synchronization intervals in the Llama sweeps. We interpret these laws through a local analysis of the behavior-dependent GRPO surrogate and a complementary mean-field model. Under local regularity conditions, the analysis yields an O(Sη) upper bound on the staleness-induced update bias that resets at synchronization. The mean-field model shows how sufficiently strong positive feedback can sustain directional drift when update directions persist across synchronization cycles. When drift speed saturates under optimizer normalization, this mechanism predicts exit from a local surrogate-validity region after an approximately fixed cumulative learning rate. Together, these findings motivate a practical calibration rule: estimate the stability threshold and collapse budget from a coarse sweep, then jointly select S and η for the intended training horizon.

Primary area:
reinforcement learning
Keywords:
asynchronous reinforcement learning, policy staleness, GRPO, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N1ToOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how policy staleness and learning rate jointly determine stability and collapse in asynchronous GRPO, without staleness corrections. Through experiments on Llama and vanilla GRPO, the paper reports two empirical scaling laws. First, the stability boundary is approximately characterized by a constant product Sη: the largest stable learning rate scales as S⁻¹ . Second, among collapsing runs, the collapse time satisfies t_obs·η = c₂ with c₂ approximately invariant to S within each Llama sweep. A conditional mean-field model connects staleness feedback, persistent drift, and AdamW normalization to the two laws. A practical rule is given that: calibrate S and η jointly from a coarse sweep, and treat long improvement under large Sη as delayed collapse rather than stability.

Critical strengths

The paper gives two clean and operationally useful scaling laws and a concrete practical payoff, and it is careful about what its evidence does and does not establish. Overall, the theoretical analysis is solid and rigorous.

Critical weaknesses

W1. Limited empirical grounding for the main scaling law.The claims regarding the general scaling laws rely on noticeably thin empirical evidence. Most configurations in the main (S,η) grid—including the 1B sweep and key stability boundary cells—are evaluated on only a single seed. Given that collapse timing is inherently heavy-tailed, observing same-batch collapse across three seeds in a isolated setup likely reflects deterministic pipeline artifacts rather than a robust physical property.Furthermore, the claim that the collapse clock is invariant to S is overextended: it rests almost entirely on Llama. For Qwen3-8B, η is varied at a single fixed S=16, which fails to prove invariance across S.

W2. The mechanism behind collapse is left a black box, so the laws may not be staleness-specific. The paper defines collapse operationally as an abrupt verifier-reward drop but never analyzes its form. No entropy, length, repetition, or KL statistics at collapse are reported. No experiments rule out alternative mechanisms, other than staleness, that could yield the same 1/η clock behavior.

Questions for authors

Q1. The three-seed repetition covers only 3B at S=32. Do the same-batch collapse regularity and the clock's S-invariance hold at S=8 and S=16, and on the 1B model? If collapse timing is seed-sensitive outside this column, how much of the t_obs·η = c₂ invariance survives?

Q2. What does collapse look like? Please report entropy, length, repetition, and KL trajectories around the collapse point. And does a no-staleness control (S=1, same η) at the collapsing learning rates fail to collapse, or collapse on a different clock? This is a better test that the clock is staleness-driven rather than a generic large-η phenomenon.

Reviewer_v8NWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the synchronization interval S and the learning rate lr influence training collapse in asynchronous GRPO. The authors find that the stability boundary is primarily determined by the product S x lr, while the time to collapse is approximately proportional to 1/lr. Based on these findings, they propose a theoretical explanation and calibration rules for practical training.

Critical strengths

The distinction between "avoiding collapse" and "merely delaying collapse" is a valuable observation.

The use of controlled experiments, extended training durations, and multi-seed validation for select configurations provides a foundation for the reported results.

The theoretical analysis offers an insightful explanation for the observed patterns.

Critical weaknesses

The practical utility of this work appears limited. The experiments are restricted to 1B, 3B, and 8B models. Given this scale, the authors should have quantified the efficiency gains provided by asynchronous training; however, the paper lacks comparisons regarding throughput or the time required to reach target reward thresholds. Furthermore, extrapolating these patterns from such small models to those with hundreds or thousands of billions of parameters is a significant leap. The intrinsic capabilities of a model, in relation to task difficulty, could fundamentally alter its training dynamics and collapse behavior.

The calibration process itself may render the approach impractical. Estimating a "collapse budget" requires observing multiple failed training runs. For extremely large models, these failures could consume substantial time and computational resources before the method can provide any meaningful guidance. The paper fails to demonstrate that the subsequent cost savings outweigh the initial calibration overhead, nor does it show whether the derived rules can accurately predict the behavior of configurations not included in the fitting process. Relying on "trial-and-error" runs that lead to collapse is not a compelling or viable solution for real-world applications.

There appears to be an inconsistency between the stated theoretical objective and the actual implementation. Equation (2) calculates the loss by first averaging over individual responses and then averaging across all responses, whereas Table 2 suggests an average taken at the token level. These two approaches assign different weights to responses of varying lengths. The authors should clarify which objective was actually employed in their experiments.

The study lacks a comparison with synchronous training. Experiments with S=1 (using the same learning rate) should be included to determine whether a similar "collapse clock" emerges even in the absence of policy lag.

Questions for authors

Under the experimental setup described in this paper, how much of an efficiency gain does asynchronous training actually provide?

When accounting for the full cost of trial runs and failed attempts, does this calibration method—compared to conventional hyperparameter tuning—actually save on computational resources or total training time?

Can the proposed rule predict the performance of configurations that were not included in the fitting process? How sensitive are these predictions to the model’s initial capabilities?

What specific loss aggregation method is used in the actual implementation? Does a run with S=1 exhibit a similar "collapse clock" (i.e., a predictable timeline leading to model collapse)?

Reviewer_xkHjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies failure in vanilla GRPO when the behavior policy is refreshed every S updates. It reports two empirical regularities: stability depends approximately on Sη, while collapse time in unstable runs scales approximately as 1/η. A local staleness bound and a conditional drift model motivate these observations. Experiments cover Llama models on mathematics and a Qwen model on DAPO.

Critical strengths
  1. The distinction between preventing collapse and merely postponing it is valuable. The extended runs are a strong part of the paper: surviving well beyond the predicted failure time makes the stable regime more credible than a short training curve would.
  2. The learning-rate sweeps reveal a surprisingly consistent collapse clock.
Critical weaknesses
  1. The collapse-time scaling is largely built into the assumptions. Assuming approximately linear drift in cumulative learning rate and collapse at a fixed exit distance makes (tcollapse∝1/ηt_{\mathrm{collapse}}\propto1/\eta) an almost immediate consequence. The theorem quantifies errors around this relationship, but offers limited independent explanation of why actual GRPO training should exhibit it.
  2. Missing low-staleness baselines. Practical asynchronous RL systems often favor small policy lags, yet the experiments omit (S=1,2,4), including the synchronous baseline. It therefore remains unclear whether the proposed scaling laws hold in these practically relevant low-staleness regimes. Including these settings would strengthen the empirical support for the paper’s practical implications.
  3. Unclear practical value of the calibration rule. Although the scaling constants are acknowledged to be setup-dependent, the robustness of the scaling relationships to changes in for example model size, training precision, data distribution, and optimization settings remains insufficiently tested. The paper also does not demonstrate that a small calibration sweep reliably predicts unseen configurations or reduces tuning costs compared with a conventional hyperparameter search.
Questions for authors

Please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.