acceptodds
Under review as a conference paper at ICLR 2027

Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO

Abstract

Production RL for language models lets the sampler fall behind the learner and repairs the resulting mismatch with a truncated importance weight. We ask how long the sampler can go without a refresh under that correction, and find a cliff: on Qwen2.5-Math-1.5B and GSM8K, importance-corrected GRPO refreshed every 192 updates learns well for 180 steps and then degrades severely in all three data seeds before the refresh arrives. Published remedies for staleness act on the update; we act on the sampler instead. Decoupled cooling draws samples at temperature 0.8 while the learner, the reference model and the importance weights stay at temperature 1, with the behaviour probability recorded from the tempered distribution, so the learner's objective is unchanged. All corresponding cooled runs are stable, and the longer interval keeps what the short one delivered: at the same update budget, a cooled sampler refreshed every 192 steps matches an uncooled sampler refreshed every 96 at the end of training (0.857 for both) and averaged over it (0.79), whereas lowering the learning rate to a safe value ends 3–7 points lower. On Qwen2.5-Math-7B the degradation points at interval 192 predict that an interval of 144 is fatal without cooling and survivable with it; on two data seeds the uncooled runs degrade before their first refresh and the cooled runs pass it and end at 92–93% against 68–81%, with one cooled run degrading transiently late in the second cycle. The benefit has a window: at three times the safe interval and in a high-mismatch MATH setting cooling delays degradation without preventing it, stronger cooling is not better, and cooling without the correction collapses. Sampling temperature is a control on staleness tolerance, and temperature and refresh interval should be chosen together.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, GRPO, asynchronous RL, off-policy, staleness, importance sampling, sampling temperature, LLM reasoning, training stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rn9WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies GRPO training with stale sampler weights and identifies a “staleness cliff”: training can initially improve and then degrade before the next sampler refresh, despite truncated importance sampling (TIS). The proposed intervention lowers the sampling temperature to 0.8, while keeping the learner temperature at 1. Experiments on Qwen2.5-Math-1.5B and 7B, using GSM8K and MATH, show that cooling reduces the measured learner–sampler mismatch and extends the tolerable refresh interval in some settings. The benefit has limits: cooling does not consistently prevent degradation at larger staleness or in the harder MATH setting.

Critical strengths

A simple intervention with a useful empirical benefit. On 1.5B/GSM8K, cooling allows a refresh interval of 192 to match the last-five-validation mean accuracy of uncooled training at 96, with both achieving 0.857 (Table 1). This is a practical result because the intervention requires little implementation effort and preserves learning progress while reducing refresh frequency.

Critical weaknesses
  1. The practical scope of the benefit remains insufficiently established. The experiments emulate staleness through fixed refresh intervals of 96–288 learner updates, without an N=1 baseline or a sweep over smaller intervals. Although the N=96 results show no accuracy penalty from cooling, they do not establish its behavior with fresh weights or small lag. Consequently, the results establish a benefit in the tested staged-training regime, but provide limited evidence for its value in practical synchronous training.
  2. The evidence for the temperature window is incomplete. Temperatures below 0.8 are mainly tested in the N=288 regimes where training already degrades, rather than in the central N=192 setting where cooling succeeds. This leaves unclear how sensitive the main result is to temperature and whether 0.8 transfers beyond the tested configurations.
Questions for authors
  1. How do T_b=0.8 and T_b=1.0 compare at N=1 and smaller refresh intervals?
  2. How was T_b=0.8 selected? Are there results for colder temperatures at N=192, and measurements of response diversity or mixed-reward groups that support the proposed explanation for why further cooling stops helping?
Reviewer_aUdqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of sampler staleness on GRPO with truncated importance sampling (TIS). Through experiments on Qwen2.5-Math-1.5B and 7B, the authors observe that long sampler refresh intervals can lead to severe training degradation, accompanied by significant learner sampler distribution mismatch. They propose decoupled cooling, which lowers the sampler temperature to 0.8 while keeping the learner temperature at 1. In their experiments, this helps prevent training collapse and achieves accuracy comparable to more frequent sampler synchronization with the same number of training updates. It also outperforms the lower learning rate baselines they tested.

Critical strengths
  1. The observed staleness cliff is an interesting finding, and the authors show that a simple cooling strategy can effectively mitigate this issue.

  2. The comparisons with lower learning rates and more frequent sampler updates are useful. Cooling achieves higher accuracy than the lower learning rate baselines tested and similar performance to more frequent synchronization, while using fewer sampler refreshes.

Critical weaknesses
  1. The paper does not sufficiently explain why cooling improves training stability. In Section 6, the authors argue that importance ratios change more slowly for tokens with higher probabilities. However, this claim is neither theoretically justified nor directly validated. Although the experiments show that cooling reduces the KL divergence between the learner and sampler, this observation does not explain why these ratios should change more slowly or how slower changes would lead to more stable training.

  2. The choice of sampling temperature is not sufficiently justified. The paper mainly uses a temperature of 0.8 but does not provide a clear principle for selecting the temperature across different settings. Moreover, it is unclear whether the improvements are specifically due to mitigating staleness or whether cooling itself produces better samples by changing the reward and advantage distributions. The paper does not sufficiently distinguish these effects.

  3. The experiments are limited to mathematical reasoning tasks (GSM8K and MATH) and a single model family (Qwen2.5-Math). It remains unclear whether the proposed cooling strategy can generalize to other types of RL tasks or model families.

Questions for authors
  1. Changing the sampler temperature affects both the importance weights and the group normalized advantages in GRPO. Could the authors clarify which of these effects is primarily responsible for the improved training stability, or whether both contribute? For example, examining the fraction of importance weights that are truncated and the distribution of group-normalized advantages at different temperatures could help clarify the underlying mechanism.

  2. How should the sampling temperature be selected for different refresh intervals? In particular, could the improvements at a temperature of 0.8 partly result from generating higher reward samples, independently of staleness? Comparing the sampled reward distributions across temperatures using the same model checkpoint could help clarify this.

Reviewer_WF98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how sampling temperature affects the tolerance of importance-corrected GRPO to a stale behavior policy.

The proposed intervention, decoupled cooling, lowers the sampler temperature while keeping the learner and reference-model likelihoods at temperature 1.

The truncated importance weights use the actual tempered behavior probabilities. The authors report that moderate cooling can delay severe training degradation and permit longer intervals between sampler refreshes. They compare against shorter refresh intervals, lower learning rates, and a port of the mu-GRPO update rule.

The main contribution is an empirical study of the interaction between sampling temperature and refresh interval.

Critical strengths
  • The intervention is simple and addresses a useful control variable in training with stale policies.

  • The experiments examine meaningful alternatives and report the method’s limits.

  • The study includes multiple seeds for the central comparisons, two model sizes, controls without importance correction, stronger cooling, and a comparison with the mu-GRPO update rule.

  • The paper also reports failures on longer intervals, difficult MATH settings, and later training cycles, as well as the lack of an additive benefit with mu-GRPO. These results help define what the intervention can and cannot achieve.

Critical weaknesses
  • The benefit at the best checkpoint is not sufficiently established relative to stable baselines. Delaying training collapse does not by itself imply that the best available checkpoint has better capabilities. The main comparisons emphasize end-of-run and whole-run accuracy, both of which are affected by late degradation. Although the paper reports peak accuracies, it does not establish a consistent advantage over stable baselines, particularly the shorter-refresh baseline, when each method is evaluated at its best checkpoint under a common selection protocol and matched training budget. Selecting checkpoints on a separate validation set and evaluating them on held-out test data would clarify how much of the benefit reflects better attainable performance and how much reflects maintaining performance later in training.

  • The model and benchmark coverage provides limited evidence of generalization to modern reasoning training. The experiments use only Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, with the central results on GSM8K and supplementary experiments on MATH. This older model family and limited set of mathematics benchmarks do not adequately establish the relevance of the findings to newer, stronger models and more challenging tasks. In particular, GSM8K consists of relatively simple arithmetic word problems; reward saturation and exploration requirements may differ substantially on harder tasks, potentially changing the interaction between temperature and staleness. The authors should test whether the same degradation pattern and cooling benefit persist with more recent base models and more challenging reasoning datasets.

Questions for authors
  • Does the advantage remain when comparing the best checkpoints? Under the same training budget and validation-based checkpoint-selection rule, how does the proposed method compare with the best checkpoints of the uncooled baseline, including those obtained before collapse, and stable baselines such as the shorter-refresh setting? Please distinguish improvements in best-checkpoint test performance from improvements in final or whole-run accuracy.

  • Do the findings generalize to newer models and harder tasks? Can the authors repeat a central temperature-versus-refresh-interval comparison using a more recent base model and a more challenging reasoning dataset? Does the same collapse pattern occur, and does cooling still help when the task's difficulty and reward saturation differ from those in the current experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.