acceptodds
Withdrawn from ICLR 2027

Random Refresh Is All You Forgot: Rethinking Staleness in Offline GRPO

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning for vision-language models is increasingly bottlenecked by rollout generation, motivating the reuse of stale, off-policy data. Existing methods address this problem through increasingly sophisticated selection, replay, reweighting, or clipping mechanisms, but rarely ask whether they outperform a much simpler alternative: refreshing a small random fraction of the rollout pool. We systematically compare eight methods that can be used to mitigate data staleness against a  15% random-refresh control across eight multimodal benchmarks. On the primary model family, none of 64 method-benchmark cells exceeds random refresh by more than 1.2 accuracy points, with a median margin of -4.2 points; 31 of 34 cross-family spot checks show the same pattern. We then ask where staleness actually concentrates. Across 26 family-benchmark cells and four model families, visually dependent tokens exhibit substantially higher staleness rates than language-prior tokens, with a mean high-|VD|/low-|VD| ratio of 9.1. Motivated by these findings, we construct a selective-refresh feasibility test that combines a small explorer with partial refresh by a larger trainee. The resulting system combines a small explorer with a larger trainee and refreshes groups with high visual dependence, supporting the first two findings while reducing the cost of full-size rollout sampling. Together, these results shift the focus from increasingly elaborate corrections of stale rollouts toward a simpler principle: maintaining freshness matters, and random refresh should be treated as a standard control whenever off-policy GRPO methods are evaluated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_txPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of data staleness in offline GRPO training for vision-language models (VLMs), where previously generated rollouts are reused across multiple optimization epochs to reduce sampling overhead. The authors systematically compare eight existing selection-based and distribution-correction methods against a simple baseline that randomly refreshes approximately 15% of the rollout pool. Their experiments suggest that random refresh is surprisingly competitive, outperforming most existing approaches across multiple multimodal benchmarks.

The paper further investigates the relationship between token-level staleness and visual dependence, reporting that visually dependent tokens exhibit substantially higher staleness rates than language-prior tokens. Motivated by this observation, the authors propose a selective-refresh strategy that uses a smaller explorer model to construct the initial rollout pool and selectively regenerates high-scoring rollout groups using a larger trainee model. Experimental results demonstrate promising accuracy–efficiency trade-offs compared with random refresh and on-policy GRPO.

Critical strengths
  1. Important and practically relevant research problem. The paper addresses rollout staleness and sampling efficiency, which are important challenges in GRPO-based post-training. The observation that a simple random-refresh baseline can outperform more sophisticated approaches is interesting and may encourage the community to reconsider how off-policy training methods are evaluated.

  2. Relatively comprehensive empirical comparisons. The authors evaluate eight representative methods across eight multimodal benchmarks on the primary model family, with additional experiments involving three other model families. This provides useful empirical evidence regarding the effectiveness of random refresh under the investigated training settings.

  3. Interesting token-level analysis. The investigation of the relationship between visual dependence and staleness provides a potentially useful perspective on the behavior of multimodal models during reinforcement learning. The reported consistency across multiple model families is encouraging.

  4. Practical consideration of sampling efficiency. The proposed small-explorer/large-trainee framework offers a reasonable approach to reducing expensive rollout generation. The paper also includes several ablation studies examining refresh ratios, rollout pool sources, and refresh-target selection strategies.

Critical weaknesses
  1. The definition of staleness is questionable and may not accurately characterize policy drift.

A major concern is the definition of token-level staleness in Section 5. Specifically, the authors classify a token as stale when its log-probability under the current policy falls below a fixed threshold of -2.5. However, this criterion measures the absolute likelihood of a token under the current policy rather than the distributional discrepancy between the behavior policy and the updated policy.

A token with a consistently low probability under both policies may be classified as stale even if its probability has increased during training. Conversely, a token whose probability decreases substantially may not be classified as stale if its current probability remains above the threshold.

Therefore, the reported correlation between visual dependence and staleness may partially reflect differences in the intrinsic probability distributions of visually dependent and language-prior tokens, rather than actual policy drift.

I strongly encourage the authors to consider more principled measurements, such as changes in token log-probability, importance-sampling ratios, or KL divergence between the behavior and current policies. It would be important to verify whether the central finding remains valid under these alternative definitions.

  1. The training scale is limited, making the generalizability of the conclusions uncertain.

Most experiments use only 200 rollout groups and eight training epochs. While this setup may be sufficient for preliminary empirical investigations, it is substantially smaller than practical large-scale reinforcement learning workloads.

The severity of policy staleness and the effectiveness of different correction strategies may change significantly with larger datasets, longer training horizons, more optimization steps, and different rollout reuse frequencies.

This limitation is particularly relevant because the paper makes broad claims regarding the effectiveness of random refresh and the inefficiency of existing methods. Larger-scale experiments or at least a systematic analysis of training scale and policy lag would considerably strengthen these conclusions.

  1. The statistical reliability of the selective-refresh improvements is not sufficiently convincing.

The paper reports that selective refresh outperforms both random refresh and on-policy GRPO in 35 out of 55 evaluated units. However, many of the reported improvements are relatively modest.

Furthermore, the evaluation relies on selecting the strongest valid seed, and the selective-refresh experiments consider multiple refresh schedules. Selecting the best-performing configuration may introduce additional selection bias, especially when the observed margins are small.

Although the authors provide additional seed-level analyses, the main conclusions would be more convincing if supported by consistent multi-seed averages, standard deviations, confidence intervals, and statistical significance tests under matched hyperparameter-selection budgets.

In particular, it is unclear whether the reported advantages would remain consistent when comparing average performance rather than the strongest observed results.

Questions for authors
  1. Why is an absolute log-probability threshold of -2.5 used to define staleness, rather than measuring the probability change between the behavior and current policies? How sensitive are the results to this threshold?
  2. How does the proposed method perform with substantially larger rollout pools, longer training schedules, or higher degrees of policy lag?
  3. Can the authors report mean performance and standard deviations across multiple seeds for the main experiments, rather than primarily relying on the strongest valid seed?
  4. Have the authors considered combining random refresh with existing methods such as SAPO, BAPO, or M2PO? Would these combinations outperform random refresh alone?
1 comment

Thank you for the time and effort you devoted to reviewing our paper. Because some review comments substantially diverge from the actual scope and content of the work, we have decided to withdraw this submission. Subject to sufficient resources, we will make every effort to clarify key information and improve the reporting of our experiments in future versions.

1. The absolute threshold and threshold sensitivity

We use this operational measure to directly examine the state of reused data under the current model; our existing threshold-sensitivity experiments support the direction of the observed relationship. Our method then uses this observation to select a lightweight refresh strategy that balances computational overhead and practical effectiveness. We faithfully report this phenomenon and its experimentally tested application, rather than claiming that the measure captures every form of policy drift.

2. Training scale

The aim of this work is to identify and validate a mechanism. The existing scale already allows us to observe the relevant trend and examine it across multiple methods, datasets, and model architectures. This breadth itself requires substantial training and evaluation. Increasing training scale across the board while maintaining that coverage would exceed the resources available to this university research project. Larger-scale experiments could extend the scope of the conclusions, but they are not a prerequisite for recognizing the phenomenon observed in the existing experiments.

3. Strongest-seed reporting and the reliability of method results

In Finding 1, we use the strongest valid seed for each existing method being compared in order to give these baselines a full opportunity to demonstrate their achievable performance. Subject to sufficient resources, we will make every effort to supplement the experiments and report multi-seed means in future versions, including the configurations and repeated-run results for our own method, so that readers can directly assess both effectiveness and stability.

4. Combining existing methods with random refresh

We have already conducted this experiment: each existing method receives the same refresh allowance and is compared directly with plain random refresh, with the results reported in the paper. Exhaustively combining seven or eight mechanisms and their configurations would quickly expand the study to hundreds of experimental configurations, exceeding its scope and available resources. However, the direct question you raised—combining an individual existing method with random refresh—is already covered by our experiments.

Reviewer_nrcnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies rollout reuse in GRPO-style training of vision-language models. It compares eight selection or distribution-correction methods with a control that randomly refreshes approximately 15% of rollout groups. It also analyzes the association between visual dependence and a token-level staleness indicator, and investigates selective refresh using a small explorer and a larger trainee. The evaluation includes multiple model families and multimodal benchmarks.

Critical strengths
  1. The paper emphasizes a simple, useful baseline and evaluates it against multiple methods and model families, rather than relying on a single favorable comparison.
  2. The selective-refresh study includes matched-refresh-fraction controls and refresh-target interventions, providing evidence beyond an uncontrolled comparison with full on-policy sampling.
Critical weaknesses
  1. The problem setting and the main comparative conclusion are not adequately aligned. Policy mismatch is an expected challenge when optimizing with previously collected rollouts; its existence does not by itself make offline learning ill-motivated. The more consequential issue is that the paper's remedy collects new rollouts, whereas the reproduced correction methods use frozen pools. Sections 3-4 and Appendix C.3 therefore compare interventions with different access to fresh data. This is informative as a practical trade-off, but it does not establish that correction mechanisms are unnecessary or inferior under a common data-access constraint. The paper should distinguish fixed-data offline learning from refresh-enabled off-policy learning, and test correction methods with the same refresh allowance. The equal-refresh targeting experiments in Section 7 do not resolve this issue for Finding 1.

  2. The staleness indicator does not isolate policy drift. Section 5 and Appendix B.1 classify a token as stale when its current log-probability is below -2.5. This does not measure a change relative to its sampling-time probability: a token can meet this condition without any policy change. Consequently, the reported high-/low-visual-dependence ratios could partly reflect differences in intrinsic token difficulty or initial likelihood, rather than differential staleness caused by training. This undermines the proposed mechanistic explanation of Finding 1. Re-evaluating probabilities every epoch is useful, but does not remove this confound from an absolute-threshold statistic.

Questions for authors
  1. Is the intended setting fixed-data offline RL or off-policy RL with additional rollout collection? When every correction method receives the same 15% refresh allowance, does random refresh without correction still dominate? Please report matched collection budgets and end-to-end compute.
  2. Does the visual-dependence result persist when staleness is measured by changes from sampling-time log-probabilities or behavior/current-policy likelihood ratios, while controlling for initial likelihood? Please include epoch-zero rates and threshold sensitivity.
  3. Can the central comparisons be reported as means and uncertainty over a predetermined common set of seeds, rather than principally the strongest valid seed per arm? How sensitive are the conclusions to this reporting choice?
1 comment

Thank you for the time and effort you devoted to reviewing our paper. Because some review comments substantially diverge from the actual scope and content of the work, we have decided to withdraw this submission. Subject to sufficient resources, we will make every effort to clarify key information and improve the reporting of our experiments in future versions.

1. Comparing frozen-data methods with refresh-enabled training

The paper's primary question is whether sophisticated data-correction mechanisms can substitute for a small amount of fresh data. Access to fresh data is therefore precisely the factor this comparison examines. In addition, experiments giving existing methods the same refresh allowance have already been conducted and reported in the main text and appendix; this material may have escaped your attention. These experiments address the supplementary question of how much additional benefit correction mechanisms provide once the same refresh allowance is available. Subject to sufficient resources, we will make every effort to present these results together more prominently in future versions.

2. Initial probability differences and training-induced drift

The paper examines the relationship between visual dependence and low-probability events under the current policy. It does not require this relationship to arise from scratch during training. Initial differences are part of our experimental setting and cannot be dismissed solely because they exist before training. Relative probability changes address a related but distinct question and do not, by themselves, invalidate the phenomenon reported here.

3. Common seed sets and reporting means

This is a mechanism-discovery study. Validation across methods, datasets, and model architectures already requires substantial training and evaluation, and the time and resources available for the original submission did not allow a uniform number of repetitions across all experiments. Subject to sufficient resources, we will make every effort to bring every method and control in Finding 1 to at least five runs and report their means and variability in future versions.

Reviewer_d214OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates rollout staleness in multimodal GRPO. It compares eight adapted methods against a baseline that randomly refreshes approximately 15% of rollout groups, analyzes the association between visual dependence and a token-level staleness measure, and evaluates selective refresh using a small explorer and a larger trainee. The main empirical claim is that inexpensive random refresh is competitive with more elaborate treatments of stale data. The selective-refresh experiments further examine whether image-related token statistics can guide which groups to regenerate.

Critical strengths

The paper raises a practically important evaluation question: whether methods for reusing stale rollouts provide benefits beyond inexpensive partial refresh. Random refresh is a simple and useful control that could change how improvements over frozen-pool baselines are interpreted. The equal-budget refresh-target intervention is also a relevant attempt to distinguish the benefit of obtaining fresh data from the benefit of choosing particular groups to refresh.

Critical weaknesses
  1. The measurements do not adequately establish the proposed mechanism. Section 5 defines staleness by thresholding the current token log-probability, without comparing it with the sampling-time probability. This can classify an unchanged, initially low-probability token as stale and therefore does not distinguish policy drift from initial unpredictability. Moreover, the refresh score in Appendix C.1 simplifies to negative log-probability under the masked image, which need not isolate visual dependence. These issues weaken both the interpretation of Finding 2 and its connection to the selective-refresh strategy.

  2. The headline comparison with random refresh is not consistently based on matched evaluation frames. Appendix A.1 identifies frame mismatches for MMStar, POPE, MME, and MathVista in Table 5, affecting 32 of the 64 displayed comparisons. The reported same-frame MathVista re-check is helpful, but a complete corrected comparison is not provided. Consequently, the headline aggregate does not yet establish the claimed pattern under consistently comparable evaluation conditions. This is consequential because the strength of random refresh relative to existing methods is the paper’s central empirical contribution.

Questions for authors
  1. Clarification of the staleness measurement: Does the implementation threshold the absolute current log-probability, as stated, or a change relative to the sampling policy? If it is the former, what evidence distinguishes training-induced drift from differences in initial token predictability?

  2. Clarification of the refresh criterion: Does “Top-VD” in Figure 5 use actual visual dependence or the masked-image surprisal score defined in Appendix C.1? What evidence supports attributing the observed benefit specifically to visual dependence rather than general token unpredictability?

  3. Evidence for the main comparison: Beyond the MathVista re-check, are matched-frame results available for the other affected benchmarks? Please provide these results, or recompute the headline aggregate using only matched-frame comparisons, so that the scope of Finding 1 can be assessed reliably.

1 comment
AuthorsOpenReview ↗

Response to Reviewer d214

Thank you for the time and effort you devoted to reviewing our paper. Because some review comments substantially diverge from the actual scope and content of the work, we have decided to withdraw this submission. Subject to sufficient resources, we will make every effort to clarify key information and improve the reporting of our experiments in future versions.

1. Definition of staleness

Our focus is the suitability of reused data under the current policy, rather than exclusively the probability changes caused by training. In our small-generator/large-trainee setting, a mismatch can exist before training begins. Therefore, observing a difference at epoch zero does not invalidate our finding. Subject to sufficient resources, we will make every effort to clarify this research objective in future versions.

2. Visual dependence and the refresh score

Finding 2 identifies highly visually dependent tokens as locations that warrant particular attention when reusing data. Motivated by this observation, our method reuses masked-image probability statistics from the small model to score rollout groups, then uses the larger model to refresh a small fraction of the highest-scoring groups. This is a practical trade-off between computational overhead and effectiveness: more elaborate measurements at every epoch could offset the time savings, whereas the feasibility of our chosen approach has been validated through direct controlled comparisons.

3. Differences in evaluation settings

Exception handling, filtering, and data-version differences led to differences in some evaluation sets. We did not repeatedly retry failed examples to obtain better scores, and we explicitly documented the differences in the paper so that readers could inspect them. More importantly, restricting the analysis to comparisons with fully matched evaluation settings still supports the conclusion that random refresh is competitive. Subject to sufficient resources, we will make every effort to standardize the reporting of these comparisons in future versions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.