acceptodds
Under review as a conference paper at ICLR 2027

ImPlay: Importance-Weighted Experience Replay for Efficient Post-Training

Abstract

Group Relative Policy Optimization (GRPO) has become a standard algorithm for post-training large language models (LLMs). However, each update depends on many on-policy rollouts, making generation the dominant training cost. Experience replay can reduce this cost by reusing past rollouts, but GRPO's group-relative advantages become stale as the policy evolves. We first analyze off-policiness in GRPO with experience replay and find that advantages can become substantially stale after only 10–20 optimizer steps, while sample age is only weakly indicative of this staleness. Motivated by this analysis, we propose ImPlay, an experience replay method for GRPO that introduces two mechanisms to address policy drift: importance-weighted advantage re-estimation, which uses importance sampling to re-estimate replayed trajectories’ group-relative advantages under the current policy; and drift-aware eviction, which uses the same signal to identify and discard trajectories that have become too off-policy for reliable reuse. Evaluated across six mathematical reasoning benchmarks, ImPlay matches GRPO's performance under the same training setup while requiring up to 47% fewer generated rollouts and 12% faster training. At a matched generation budget, ImPlay outperforms ExGRPO, the state-of-the-art experience-replay method for GRPO, by up to 3.0 percentage points on the more challenging benchmarks. Ablations show that drift-aware eviction maintains a replay buffer in which importance-weighted advantage re-estimation remains effective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Posttraining, RL, GRPO, Experience Replay

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PxqNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Based on the observation that group-based advantage gets stable after 10-20 optimization steps on math domains, this paper introduces ImPlay, which is a GRPO-based experience replay algorithm incorporating an importance-weighted advantage re-estimation and drift-aware eviction.

Critical strengths
  • Analysis in Section.4 shows that while advantages become stale rapidly as training progresses, the number of elapsed steps alone cannot accurately gauge staleness, and age-based eviction risks discarding useful experiences. This clearly motivates ImPlay.
  • Unlike existing experience replay algorithm ExGRPO, ImPlay corrects saved rollouts with importance-weights without on-policy rollouts. As shown in Table.1 ImPlay achieves comparable performance to ExGRPO with fewer rollouts (Fig.2).
Critical weaknesses
  • Advantage staleness analysis and main experiments are only on math domains. I would be willing to raise my score if the authors show consistent results (at least one additional domain) on other LLM domains (eg, code, science), or VLM domains (eg, math). Additionally, in Table.2, the baselines are only GRPO and ExGRPO. It would be better to include at least one more additional baseline (eg, ReMix, POPO,NexGRPO).
  • I consider this paper an incremental improvement over existing algorithms (eg, ExGRPO), a heuristic approach reliant on hyperparameters such as τ\tau, cc, and CC. Given the lack of theoretical guarantees, its utility must be demonstrated empirically across multiple domains, also including extensive sensitivity analyses or discussions on hyperparameters.
Questions for authors
  • I see sensitivity analysis for τ\tau, cc in Appendix C.3, but is it possible to estimate them with few steps run?
  • Did you use any RL framework (eg, verl) to conduct experiments?
  • Could the authors report or discuss additional memory or computation costs compared to ExGRPO? Especially, I'm interested in how the log-probs of past rollouts accumulate.
5 comments

We thank reviewer PxqN for highlighting the importance of our advantage staleness analysis and recognizing the efficiency of ImPlay. We address their raised concerns and questions below.

Distinction from ExGRPO and methodological novelty

We would like to clarify that ImPlay differs from ExGRPO in two core aspects of experience replay: sample eviction and advantage estimation.

First, ExGRPO uses a success-based eviction criterion, removing replay samples only when they reach a 100% success rate. In contrast, ImPlay introduces a staleness-based eviction criterion, using the current-to-behavior policy log-probability ratio to identify and evict samples with excessive policy drift.

Second, ImPlay introduces importance-weighted re-estimation of replay samples' advantages, rather than adopting ExGRPO's advantage estimation strategy with fresh generations. These two components are complementary: the eviction criterion filters out samples with excessive policy drift, while advantage re-estimation accounts for policy drift among retained samples. Putting these together, ImPlay yields considerable improvements in training efficiency, as demonstrated empirically in Figure 2.

Thus, ImPlay is not simply ExGRPO with a modified heuristic. It independently redesigns two core, complementary components of experience replay without adopting any of ExGRPO's replay-specific mechanisms.

ImPlay’s performance on other domains beyond math.

We agree that evaluation on additional domains would further strengthen the empirical evidence. We chose mathematical reasoning as our evaluation domain because it is a well-established testbed for reasoning-oriented RL, where relatively small models such as Qwen3-4B can achieve meaningful performance while leaving room for improvement, allowing us to study the effects of experience replay.

None of ImPlay's replay mechanisms are specific to mathematical reasoning. Both the staleness-based eviction criterion and importance-weighted advantage re-estimation operate on policy probabilities and rollout-level rewards, without relying on domain-specific properties. While this makes ImPlay applicable in principle to other domains, we acknowledge that its empirical effectiveness beyond mathematics remains to be validated.

RL framework

We used the verl framework to conduct our experiments.

Discussion of hyperparameters and their sensitivity analysis

c controls how strongly the importance weights adjust the baseline. The length normalization prevents the long rollouts from dominating the baseline re-estimation but can lead to close to uniform weights and thus negligible baseline correction. In our experiments, a low c=256 led to performance degradation, while both 512 and 1024 achieved competitive performance.

tau controls replay staleness. Smaller values enforce stricter filtering risking low number of samples in the buffer which in turn reduces diversity of replayed samples. Larger values of tau allow more drift before eviction risking replaying too stale samples. Among the values we tried, the impact of c is much higher than tau.

The table below shows the comparison across different values of c and tau from step 50 all the way to 500. The competitive setups are recognizable from step 300.

ctau50100150200300400500
2560.001007.818.4710.6311.6311.2812.5311.81
5120.000757.719.2011.2212.8517.8817.6717.67
5120.001007.539.2412.8117.1219.3818.5119.97
5120.001507.649.8611.9111.5618.6818.8919.51
10240.001008.549.8312.4714.5120.3819.6518.44
Reviewer_8MuAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

IMPLAY addresses stale advantages in GRPO replay by re-estimating stored rollout-group baselines with likelihood-based weights and evicting samples based on policy drift. Experiments on Qwen and OLMo evaluate accuracy, rollout usage, and training time.

Critical strengths

Independent success-rate estimates show how stored advantages become stale and why sample age alone is insufficient for deciding which trajectories to retain.

Critical weaknesses
  1. Performance under continued training is unclear. At 500 steps, IMPLAY trails ExGRPO in Table 1, particularly on OLMo at 34.47 versus 36.68. Figure 2 shows an advantage at matched rollout budgets, but also an earlier plateau on OLMo. It remains unclear whether longer training can close this gap or whether replay limits further improvement.
  2. The accuracy of baseline correction is not directly measured. The appendix reports weight diagnostics and changes in advantages, but neither establishes whether the corrected baseline is closer to the current policy's success rate than the stored baseline.
Questions for authors
  1. How many GPU-hours does each method need to reach common accuracy targets on both models?

  2. Using many fresh current-policy rollouts on sampled replay prompts, can you compare the absolute errors of stored and corrected baselines?

2 comments

We thank reviewer 8MuA for recognizing the importance of our replay advantage staleness analysis and we address their raised concerns and questions below.

Performance under continued training, and efficiency comparison on common accuracy targets.

With 1000 training steps, the performance for Qwen3-4B-Base plateaus for all methods at around 102,400 rollouts. For Olmo3-7B-Instricy-SFT, the performance of all methods continued to increase with ImPlay outperforming both GPRO baselines. ImPlay offers considerable savings at higher accuracy targets, particularly on OLMo, where it reaches 30% accuracy with 41.27% fewer GPU-hours compared to GRPO with batch size 16.

The table below compares these methods at matched rollouts budgets averaged over the three AIME benchmarks. On Qwen3-4B-Base ImPlay is ahead of GRPO baselines up to the 76,800 rollouts where GRPO with batch size 8 catches up and outperforms. All three methods converge to 20% accuracy around 102,400 rollouts. On OLMo-3-7B-Instruct-SFT, ImPlay consistently matches and outperforms both baselines and reaches the highest performance at 128000 rollouts.

ModelGenerated rolloutsGRPO (BS8)GRPO (BS16)ImPlay
Qwen3-4B-Base25,60011.158.3316.11
Qwen3-4B-Base51,20014.5811.8417.57
Qwen3-4B-Base76,80020.3518.5418.68
Qwen3-4B-Base102,40020.5620.1020.17
Qwen3-4B-Base128,00020.7320.8020.59
OLMo-3-7B-Instruct-SFT25,6009.937.229.10
OLMo-3-7B-Instruct-SFT51,20021.3511.8723.44
OLMo-3-7B-Instruct-SFT76,80025.0022.0832.36
OLMo-3-7B-Instruct-SFT102,40028.8524.5534.38
OLMo-3-7B-Instruct-SFT128,00030.0726.2235.80

The table below shows the number of GPU hours each method needs to reach common accuracy targets. Across most accuracy targets, ImPlay is more efficient than both GRPO baselines. For Qwen3-4B-Base, ImPlay reaches the highest common accuracy of 20% with 5.31% fewer GPU hours compared to GRPO with batch size 16. On OLMo-3-7B-Instruct-SFT, the highest common accuracy target is 30% which is reached by ImPlay with 41.27% fewer GPU hours than GRPO with batch size 16.

ModelTarget accuracyGRPO (BS8)GRPO (BS16)ImPlay
Qwen3-4B-Base12%17.39 (250)18.33 (210)13.24 (150)
Qwen3-4B-Base15%32.19 (430)19.42 (220)16.53 (180)
Qwen3-4B-Base18%42.54 (550)21.95 (240)25.17 (240)
Qwen3-4B-Base20%48.20 (600)43.67 (390)41.35 (350)
OLMo-3-7B-Instruct-SFT20%29.53 (300)29.01 (240)38.18 (330)
OLMo-3-7B-Instruct-SFT25%76.85 (580)72.80 (420)52.34 (400)
OLMo-3-7B-Instruct-SFT30%158.93 (980)130.83 (630)76.84 (500)
OLMo-3-7B-Instruct-SFT35%Not reachedNot reached159.28 (790)

ImPlay’s baseline correction accuracy.

The table below shows the estimated baseline absolute error for the stored baselines and ImPlay baseline across different replay age bins. ImPlay consistently provides a lower error, especially as replay samples get older.

Sample ageBirth mean absolute errorImPlay mean absolute errorError reduction
1–100.03890.03841.4%
11–200.06520.06037.4%
21–400.09650.085311.7%
41–700.14920.134010.2%

Computing ImPlay’s baseline with 1024 rollouts throughout training would require considerable memory and compute resources. As a result, we reduce the number of rollouts to 128 for this experiment.

Reviewer_1vhbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes IMPLAY, an experience replay method for GRPO that aims to tackle online generation efficiency. The method uses importance sampling to re-estimate the advantage under the current policy to both change the gradient update and to remove less reliable samples from the replay buffer.

Critical strengths

Interesting problem setting, studying how to do replay well to be more generation efficient is an interesting problem.

Critical weaknesses

Organizational structure is a bit weird and redundant. For example, Eq 6 is a choice of how to do experience replay, not background? Furthermore, related work seems to just repeat a lot of the material in the background and intro.

Language is odd and hard to follow. For example, "birth time". Writing is also a bit unnecessarily complex. For example, Eq in lines 201-202 basically says to subtract the pass rates since r=1 by construction. Unless p^\hat{p} is actually supposed to be ρ\rho? As another example, what are "shadow-replay observations"??

Analysis is a bit weak. For example, Section 4 does a large empirical study on "staleness" of samples and shows that optimizer steps induce staleness / off-policyness. But it is essentially just showing that the pass rate of prompts increases during training, which is exactly what RL aims to do? I also found the results section hard to parse. It just states things like, the mean value is -0.058, the mean drops to -0.087.

In general, the paper is dense and extremely difficult to read, so I cannot follow the argumentation. As such, I cannot provide a detailed review and recommend rejection because of poor intelligibility.

Questions for authors

N/a

1 comment

We thank reviewer 1vhb’s for their feedback on the presentation clarity and clarify the analysis in Section 4 below.

Section 4’s analysis aims to assess:

  • How quickly birth-time (generation-time) advantages become stale as the policy improves during RL post-training.
  • Is the replay sample’s age a reliable indicator of its advantage staleness.

While increasing pass rates are an expected outcome of successful RL training, our analysis concerns the implications of this improvement for the validity of previously computed advantages when using experience replay. Moreover, our analysis examines if sample age reliably predicts the advantage of staleness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.