- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies GRPO training with stale sampler weights and identifies a “staleness cliff”: training can initially improve and then degrade before the next sampler refresh, despite truncated importance sampling (TIS). The proposed intervention lowers the sampling temperature to 0.8, while keeping the learner temperature at 1. Experiments on Qwen2.5-Math-1.5B and 7B, using GSM8K and MATH, show that cooling reduces the measured learner–sampler mismatch and extends the tolerable refresh interval in some settings. The benefit has limits: cooling does not consistently prevent degradation at larger staleness or in the harder MATH setting.
Critical strengths
A simple intervention with a useful empirical benefit. On 1.5B/GSM8K, cooling allows a refresh interval of 192 to match the last-five-validation mean accuracy of uncooled training at 96, with both achieving 0.857 (Table 1). This is a practical result because the intervention requires little implementation effort and preserves learning progress while reducing refresh frequency.
Critical weaknesses
- The practical scope of the benefit remains insufficiently established. The experiments emulate staleness through fixed refresh intervals of 96–288 learner updates, without an N=1 baseline or a sweep over smaller intervals. Although the N=96 results show no accuracy penalty from cooling, they do not establish its behavior with fresh weights or small lag. Consequently, the results establish a benefit in the tested staged-training regime, but provide limited evidence for its value in practical synchronous training.
- The evidence for the temperature window is incomplete. Temperatures below 0.8 are mainly tested in the N=288 regimes where training already degrades, rather than in the central N=192 setting where cooling succeeds. This leaves unclear how sensitive the main result is to temperature and whether 0.8 transfers beyond the tested configurations.
Questions for authors
- How do T_b=0.8 and T_b=1.0 compare at N=1 and smaller refresh intervals?
- How was T_b=0.8 selected? Are there results for colder temperatures at N=192, and measurements of response diversity or mixed-reward groups that support the proposed explanation for why further cooling stops helping?