- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how policy staleness and learning rate jointly determine stability and collapse in asynchronous GRPO, without staleness corrections. Through experiments on Llama and vanilla GRPO, the paper reports two empirical scaling laws. First, the stability boundary is approximately characterized by a constant product Sη: the largest stable learning rate scales as S⁻¹ . Second, among collapsing runs, the collapse time satisfies t_obs·η = c₂ with c₂ approximately invariant to S within each Llama sweep. A conditional mean-field model connects staleness feedback, persistent drift, and AdamW normalization to the two laws. A practical rule is given that: calibrate S and η jointly from a coarse sweep, and treat long improvement under large Sη as delayed collapse rather than stability.
Critical strengths
The paper gives two clean and operationally useful scaling laws and a concrete practical payoff, and it is careful about what its evidence does and does not establish. Overall, the theoretical analysis is solid and rigorous.
Critical weaknesses
W1. Limited empirical grounding for the main scaling law.The claims regarding the general scaling laws rely on noticeably thin empirical evidence. Most configurations in the main (S,η) grid—including the 1B sweep and key stability boundary cells—are evaluated on only a single seed. Given that collapse timing is inherently heavy-tailed, observing same-batch collapse across three seeds in a isolated setup likely reflects deterministic pipeline artifacts rather than a robust physical property.Furthermore, the claim that the collapse clock is invariant to S is overextended: it rests almost entirely on Llama. For Qwen3-8B, η is varied at a single fixed S=16, which fails to prove invariance across S.
W2. The mechanism behind collapse is left a black box, so the laws may not be staleness-specific. The paper defines collapse operationally as an abrupt verifier-reward drop but never analyzes its form. No entropy, length, repetition, or KL statistics at collapse are reported. No experiments rule out alternative mechanisms, other than staleness, that could yield the same 1/η clock behavior.
Questions for authors
Q1. The three-seed repetition covers only 3B at S=32. Do the same-batch collapse regularity and the clock's S-invariance hold at S=8 and S=16, and on the 1B model? If collapse timing is seed-sensitive outside this column, how much of the t_obs·η = c₂ invariance survives?
Q2. What does collapse look like? Please report entropy, length, repetition, and KL trajectories around the collapse point. And does a no-staleness control (S=1, same η) at the collapsing learning rates fail to collapse, or collapse on a different clock? This is a better test that the clock is staleness-driven rather than a generic large-η phenomenon.