- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Based on the observation that group-based advantage gets stable after 10-20 optimization steps on math domains, this paper introduces ImPlay, which is a GRPO-based experience replay algorithm incorporating an importance-weighted advantage re-estimation and drift-aware eviction.
Critical strengths
- Analysis in Section.4 shows that while advantages become stale rapidly as training progresses, the number of elapsed steps alone cannot accurately gauge staleness, and age-based eviction risks discarding useful experiences. This clearly motivates ImPlay.
- Unlike existing experience replay algorithm ExGRPO, ImPlay corrects saved rollouts with importance-weights without on-policy rollouts. As shown in Table.1 ImPlay achieves comparable performance to ExGRPO with fewer rollouts (Fig.2).
Critical weaknesses
- Advantage staleness analysis and main experiments are only on math domains. I would be willing to raise my score if the authors show consistent results (at least one additional domain) on other LLM domains (eg, code, science), or VLM domains (eg, math). Additionally, in Table.2, the baselines are only GRPO and ExGRPO. It would be better to include at least one more additional baseline (eg, ReMix, POPO,NexGRPO).
- I consider this paper an incremental improvement over existing algorithms (eg, ExGRPO), a heuristic approach reliant on hyperparameters such as , , and . Given the lack of theoretical guarantees, its utility must be demonstrated empirically across multiple domains, also including extensive sensitivity analyses or discussions on hyperparameters.
Questions for authors
- I see sensitivity analysis for , in Appendix C.3, but is it possible to estimate them with few steps run?
- Did you use any RL framework (eg, verl) to conduct experiments?
- Could the authors report or discuss additional memory or computation costs compared to ExGRPO? Especially, I'm interested in how the log-probs of past rollouts accumulate.
5 comments
ImPlay’s performance on other domains beyond math.
We agree that evaluation on additional domains would further strengthen the empirical evidence. We chose mathematical reasoning as our evaluation domain because it is a well-established testbed for reasoning-oriented RL, where relatively small models such as Qwen3-4B can achieve meaningful performance while leaving room for improvement, allowing us to study the effects of experience replay.
None of ImPlay's replay mechanisms are specific to mathematical reasoning. Both the staleness-based eviction criterion and importance-weighted advantage re-estimation operate on policy probabilities and rollout-level rewards, without relying on domain-specific properties. While this makes ImPlay applicable in principle to other domains, we acknowledge that its empirical effectiveness beyond mathematics remains to be validated.
RL framework
We used the verl framework to conduct our experiments.
Discussion of hyperparameters and their sensitivity analysis
c controls how strongly the importance weights adjust the baseline. The length normalization prevents the long rollouts from dominating the baseline re-estimation but can lead to close to uniform weights and thus negligible baseline correction. In our experiments, a low c=256 led to performance degradation, while both 512 and 1024 achieved competitive performance.
tau controls replay staleness. Smaller values enforce stricter filtering risking low number of samples in the buffer which in turn reduces diversity of replayed samples. Larger values of tau allow more drift before eviction risking replaying too stale samples. Among the values we tried, the impact of c is much higher than tau.
The table below shows the comparison across different values of c and tau from step 50 all the way to 500. The competitive setups are recognizable from step 300.
| c | tau | 50 | 100 | 150 | 200 | 300 | 400 | 500 |
|---|---|---|---|---|---|---|---|---|
| 256 | 0.00100 | 7.81 | 8.47 | 10.63 | 11.63 | 11.28 | 12.53 | 11.81 |
| 512 | 0.00075 | 7.71 | 9.20 | 11.22 | 12.85 | 17.88 | 17.67 | 17.67 |
| 512 | 0.00100 | 7.53 | 9.24 | 12.81 | 17.12 | 19.38 | 18.51 | 19.97 |
| 512 | 0.00150 | 7.64 | 9.86 | 11.91 | 11.56 | 18.68 | 18.89 | 19.51 |
| 1024 | 0.00100 | 8.54 | 9.83 | 12.47 | 14.51 | 20.38 | 19.65 | 18.44 |
We thank reviewer PxqN for highlighting the importance of our advantage staleness analysis and recognizing the efficiency of ImPlay. We address their raised concerns and questions below.
Distinction from ExGRPO and methodological novelty
We would like to clarify that ImPlay differs from ExGRPO in two core aspects of experience replay: sample eviction and advantage estimation.
First, ExGRPO uses a success-based eviction criterion, removing replay samples only when they reach a 100% success rate. In contrast, ImPlay introduces a staleness-based eviction criterion, using the current-to-behavior policy log-probability ratio to identify and evict samples with excessive policy drift.
Second, ImPlay introduces importance-weighted re-estimation of replay samples' advantages, rather than adopting ExGRPO's advantage estimation strategy with fresh generations. These two components are complementary: the eviction criterion filters out samples with excessive policy drift, while advantage re-estimation accounts for policy drift among retained samples. Putting these together, ImPlay yields considerable improvements in training efficiency, as demonstrated empirically in Figure 2.
Thus, ImPlay is not simply ExGRPO with a modified heuristic. It independently redesigns two core, complementary components of experience replay without adopting any of ExGRPO's replay-specific mechanisms.