- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies zero-variance groups in GRPO-style search-agent training, where all-correct or all-incorrect rollout groups provide no group-relative gradient. The key observation is that zero-variance is policy-dependent and temporally non-stationary: a query that is too hard or too easy at one training step may later become signal-bearing as the policy improves or drifts. Based on this, the authors propose query recycling, which removes queries once they contribute an effective gradient update but keeps zero-variance queries in the active pool for future resampling. Experiments show the effectiveness of the proposed method compared with matched-budget GRPO/Bounded-DAPO baselines.
Critical strengths
The paper is overall well written.
The proposed method is simple but seemingly effective.
Experiments are conducted to validate the proposed method.
Critical weaknesses
The statement in the abstract that “existing approaches treat zero-variance as a static property” seems overly broad. The prior work DAVID-GRPO already assumes that an all-failed group may become solvable under additional resampling. This raises a slight novelty concern, as the proposed method is very simple.
The too-easy-only and too-hard-only ablation is only conducted on the 1.7B model.
The reported error bars appear to reflect repeated inference runs rather than independent RL training seeds. Given that the gains from recycling are relatively modest (roughly 1–2 Pass@1 points), it would be useful to report results across multiple training seeds to establish that the improvements are robust to RL training variance.
Questions for authors
According to Table1, on the 1.7B model, the too-easy-only recycling variant provides a larger gain than the too-hard-only variant, which is somewhat counter-intuitive given that model capability is generally expected to improve during training. Can the authors disentangle genuine policy drift/forgetting from finite- K K misclassification as the source of too-easy rescues, and report the same too-easy-only versus too-hard-only ablation for the 4B model to verify whether this effect persists at larger scale?
Since smaller group sizes can substantially increase apparent recycling through finite-sample misclassification, could the authors report a matched-budget comparison between recycling and no-recycling at K
8 K=8 and possibly further enlarge the value of K K?
Could the authors report repeated experiments with independent RL training seeds?
3 comments
Thank you for the rebuttal. It addresses several of my concerns.
However, the K=8 gain is modest (+0.30), independent RL training seeds are still missing, and the 4B too-easy/too-hard ablation remains incomplete. The relative contributions of finite-K noise and policy changes also remain unclear.
Overall, the rebuttal strengthens the paper, but some empirical concerns remain. Currently, I maintain my original rating of 3.
Answer
Thanks for the follow-up, and for stating that the rebuttal solved several of your concerns.
The 4B too-hard-only arm has finished under a training setup that fully matches the paper:
| Method | 2Wiki | Bamboogle | HotpotQA | MuSiQue | NQ | PopQA | TriviaQA | Avg |
|---|---|---|---|---|---|---|---|---|
| 4B recycle all | 79.1 | 84.0 | 74.4 | 41.3 | 74.4 | 55.8 | 89.9 | 71.3 |
| 4B too-hard only | 78.5 | 80.0 | 73.4 | 42.0 | 72.1 | 51.6 | 91.2 | 69.8 |
While unfortunatelly the too-easy-only arm is still running, current results are consistent with the ablation at the 1.4B scale, where recycling both too-easy and too-hard brings the strongest improvement.
As for the reviewer’s remaining points:
- The K=8 experiment (N1, global answer) was conducted on a single training epoch to fit the rebuttal’s timeframe.
- Regarding finite-K, we note that experiment N2.1 compares a setting where an arm only solves via finite-K noise mitigation (i.e., increasing K under the same policy), while the other only solves via policy change (i.e., deferring 4 rollouts to a future policy), and the latter achieved stronger performance.
Answer
We thank the reviewer for the comments. Please find below our responses to each of the raised points.
W1 - The “prior work treats zero-variance as a static property” framing
We agree with the reviewer that some previous studies already refrain from treating zero-variance as a static property, namely the DAVID-GRPO paper, and the ERPO paper brought to our attention by reviewer ZSXm. We will revise our framing and positioning, by stating that (i) those methods try to exploit the current policy that led to a zero-variance group, while we offload to a future policy, and (ii) we focus on both too-easy and too-hard queries, while the methods above target only one of these groups.
Q1 - K=8 and finite sampling noise, too-easy vs too-hard
We agree that the signal recycling recovers can come from two sources, i.e. policy change between visits, or finite-K sampling noise. We also agree that the paper does not separate them, and we will make this explicit. The subset analysis from N1 (global response) does so under a fixed policy. Any 4 of the 8 rollouts in a K=8 group are what a K=4 group would have produced, so enumerating all C(8,4) subsets tells us, for each group, how often K=4 would have reported a wrong label. Two views of the same data follow:
On a matched budget, N1 compares recycling and no-recycling at K=8. Raising K uniformly is costly, since every query pays for the larger group regardless of need. Recycling can be seen as an adaptive increase in K across policy updates (N2.1), so each query consumes only the rollouts it needs to yield a training signal.
W2, W3, Q2 - Ablations at the 4B parameter scale and multiple seeds
We agree these experiments would strengthen the empirical case. We have started the too-easy-only VS too-hard-only ablation at the 4B parameter scale. We will update both this thread and the global answer with any experiments that finish within the rebuttal window, and a revised version of the manuscript can include the results.