- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of data staleness in offline GRPO training for vision-language models (VLMs), where previously generated rollouts are reused across multiple optimization epochs to reduce sampling overhead. The authors systematically compare eight existing selection-based and distribution-correction methods against a simple baseline that randomly refreshes approximately 15% of the rollout pool. Their experiments suggest that random refresh is surprisingly competitive, outperforming most existing approaches across multiple multimodal benchmarks.
The paper further investigates the relationship between token-level staleness and visual dependence, reporting that visually dependent tokens exhibit substantially higher staleness rates than language-prior tokens. Motivated by this observation, the authors propose a selective-refresh strategy that uses a smaller explorer model to construct the initial rollout pool and selectively regenerates high-scoring rollout groups using a larger trainee model. Experimental results demonstrate promising accuracy–efficiency trade-offs compared with random refresh and on-policy GRPO.
Critical strengths
-
Important and practically relevant research problem. The paper addresses rollout staleness and sampling efficiency, which are important challenges in GRPO-based post-training. The observation that a simple random-refresh baseline can outperform more sophisticated approaches is interesting and may encourage the community to reconsider how off-policy training methods are evaluated.
-
Relatively comprehensive empirical comparisons. The authors evaluate eight representative methods across eight multimodal benchmarks on the primary model family, with additional experiments involving three other model families. This provides useful empirical evidence regarding the effectiveness of random refresh under the investigated training settings.
-
Interesting token-level analysis. The investigation of the relationship between visual dependence and staleness provides a potentially useful perspective on the behavior of multimodal models during reinforcement learning. The reported consistency across multiple model families is encouraging.
-
Practical consideration of sampling efficiency. The proposed small-explorer/large-trainee framework offers a reasonable approach to reducing expensive rollout generation. The paper also includes several ablation studies examining refresh ratios, rollout pool sources, and refresh-target selection strategies.
Critical weaknesses
- The definition of staleness is questionable and may not accurately characterize policy drift.
A major concern is the definition of token-level staleness in Section 5. Specifically, the authors classify a token as stale when its log-probability under the current policy falls below a fixed threshold of -2.5. However, this criterion measures the absolute likelihood of a token under the current policy rather than the distributional discrepancy between the behavior policy and the updated policy.
A token with a consistently low probability under both policies may be classified as stale even if its probability has increased during training. Conversely, a token whose probability decreases substantially may not be classified as stale if its current probability remains above the threshold.
Therefore, the reported correlation between visual dependence and staleness may partially reflect differences in the intrinsic probability distributions of visually dependent and language-prior tokens, rather than actual policy drift.
I strongly encourage the authors to consider more principled measurements, such as changes in token log-probability, importance-sampling ratios, or KL divergence between the behavior and current policies. It would be important to verify whether the central finding remains valid under these alternative definitions.
- The training scale is limited, making the generalizability of the conclusions uncertain.
Most experiments use only 200 rollout groups and eight training epochs. While this setup may be sufficient for preliminary empirical investigations, it is substantially smaller than practical large-scale reinforcement learning workloads.
The severity of policy staleness and the effectiveness of different correction strategies may change significantly with larger datasets, longer training horizons, more optimization steps, and different rollout reuse frequencies.
This limitation is particularly relevant because the paper makes broad claims regarding the effectiveness of random refresh and the inefficiency of existing methods. Larger-scale experiments or at least a systematic analysis of training scale and policy lag would considerably strengthen these conclusions.
- The statistical reliability of the selective-refresh improvements is not sufficiently convincing.
The paper reports that selective refresh outperforms both random refresh and on-policy GRPO in 35 out of 55 evaluated units. However, many of the reported improvements are relatively modest.
Furthermore, the evaluation relies on selecting the strongest valid seed, and the selective-refresh experiments consider multiple refresh schedules. Selecting the best-performing configuration may introduce additional selection bias, especially when the observed margins are small.
Although the authors provide additional seed-level analyses, the main conclusions would be more convincing if supported by consistent multi-seed averages, standard deviations, confidence intervals, and statistical significance tests under matched hyperparameter-selection budgets.
In particular, it is unclear whether the reported advantages would remain consistent when comparing average performance rather than the strongest observed results.
Questions for authors
- Why is an absolute log-probability threshold of -2.5 used to define staleness, rather than measuring the probability change between the behavior and current policies? How sensitive are the results to this threshold?
- How does the proposed method perform with substantially larger rollout pools, longer training schedules, or higher degrees of policy lag?
- Can the authors report mean performance and standard deviations across multiple seeds for the main experiments, rather than primarily relying on the strongest valid seed?
- Have the authors considered combining random refresh with existing methods such as SAPO, BAPO, or M2PO? Would these combinations outperform random refresh alone?
Thank you for the time and effort you devoted to reviewing our paper. Because some review comments substantially diverge from the actual scope and content of the work, we have decided to withdraw this submission. Subject to sufficient resources, we will make every effort to clarify key information and improve the reporting of our experiments in future versions.
1. The absolute threshold and threshold sensitivity
We use this operational measure to directly examine the state of reused data under the current model; our existing threshold-sensitivity experiments support the direction of the observed relationship. Our method then uses this observation to select a lightweight refresh strategy that balances computational overhead and practical effectiveness. We faithfully report this phenomenon and its experimentally tested application, rather than claiming that the measure captures every form of policy drift.
2. Training scale
The aim of this work is to identify and validate a mechanism. The existing scale already allows us to observe the relevant trend and examine it across multiple methods, datasets, and model architectures. This breadth itself requires substantial training and evaluation. Increasing training scale across the board while maintaining that coverage would exceed the resources available to this university research project. Larger-scale experiments could extend the scope of the conclusions, but they are not a prerequisite for recognizing the phenomenon observed in the existing experiments.
3. Strongest-seed reporting and the reliability of method results
In Finding 1, we use the strongest valid seed for each existing method being compared in order to give these baselines a full opportunity to demonstrate their achievable performance. Subject to sufficient resources, we will make every effort to supplement the experiments and report multi-seed means in future versions, including the configurations and repeated-run results for our own method, so that readers can directly assess both effectiveness and stability.
4. Combining existing methods with random refresh
We have already conducted this experiment: each existing method receives the same refresh allowance and is compared directly with plain random refresh, with the results reported in the paper. Exhaustively combining seven or eight mechanisms and their configurations would quickly expand the study to hundreds of experimental configurations, exceeding its scope and available resources. However, the direct question you raised—combining an individual existing method with random refresh—is already covered by our experiments.