acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Zero-Variance Groups via Query Recycling for Agentic Search Training

Abstract

The use of GRPO-style algorithms has become the standard strategy for training LLM search agents under outcome-only rewards. With these algorithms, a query contributes to parameter updates only when its rollout group mixes successes and failures; all-correct (too-easy) and all-incorrect (too-hard) groups are zero-variance and waste rollout cost. Existing approaches treat zero-variance as a static property and either discard or pre-filter such groups. We hypothesize and empirically validate that queries flip between zero-variance and signal-bearing states as the policy evolves during training. Building on this intuition, we propose query recycling, which returns zero-variance groups to a mutable pool for future resampling, so that the effective training distribution co-evolves with the policy. Using the proposed technique, a 4B parameter model trained exclusively on synthetic data achieves 71.3 average Pass@1 across seven multi-hop QA benchmarks, outperforming prior systems trained on benchmark-derived supervision. Analysis of recycling patterns shows that recycled queries supply roughly three quarters of the effective batch by the end of training, with contributions split between recovery from policy improvement and policy drift.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Search, GRPO, Zero-Variance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X3ZhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies zero-variance groups in GRPO-style search-agent training, where all-correct or all-incorrect rollout groups provide no group-relative gradient. The key observation is that zero-variance is policy-dependent and temporally non-stationary: a query that is too hard or too easy at one training step may later become signal-bearing as the policy improves or drifts. Based on this, the authors propose query recycling, which removes queries once they contribute an effective gradient update but keeps zero-variance queries in the active pool for future resampling. Experiments show the effectiveness of the proposed method compared with matched-budget GRPO/Bounded-DAPO baselines.

Critical strengths

The paper is overall well written.

The proposed method is simple but seemingly effective.

Experiments are conducted to validate the proposed method.

Critical weaknesses

The statement in the abstract that “existing approaches treat zero-variance as a static property” seems overly broad. The prior work DAVID-GRPO already assumes that an all-failed group may become solvable under additional resampling. This raises a slight novelty concern, as the proposed method is very simple.

The too-easy-only and too-hard-only ablation is only conducted on the 1.7B model.

The reported error bars appear to reflect repeated inference runs rather than independent RL training seeds. Given that the gains from recycling are relatively modest (roughly 1–2 Pass@1 points), it would be useful to report results across multiple training seeds to establish that the improvements are robust to RL training variance.

Questions for authors

According to Table1, on the 1.7B model, the too-easy-only recycling variant provides a larger gain than the too-hard-only variant, which is somewhat counter-intuitive given that model capability is generally expected to improve during training. Can the authors disentangle genuine policy drift/forgetting from finite- K K misclassification as the source of too-easy rescues, and report the same too-easy-only versus too-hard-only ablation for the 4B model to verify whether this effect persists at larger scale?

Since smaller group sizes can substantially increase apparent recycling through finite-sample misclassification, could the authors report a matched-budget comparison between recycling and no-recycling at K

8 K=8 and possibly further enlarge the value of K K?

Could the authors report repeated experiments with independent RL training seeds?

3 comments

We thank the reviewer for the comments. Please find below our responses to each of the raised points.

W1 - The “prior work treats zero-variance as a static property” framing

We agree with the reviewer that some previous studies already refrain from treating zero-variance as a static property, namely the DAVID-GRPO paper, and the ERPO paper brought to our attention by reviewer ZSXm. We will revise our framing and positioning, by stating that (i) those methods try to exploit the current policy that led to a zero-variance group, while we offload to a future policy, and (ii) we focus on both too-easy and too-hard queries, while the methods above target only one of these groups.

Q1 - K=8 and finite sampling noise, too-easy vs too-hard

We agree that the signal recycling recovers can come from two sources, i.e. policy change between visits, or finite-K sampling noise. We also agree that the paper does not separate them, and we will make this explicit. The subset analysis from N1 (global response) does so under a fixed policy. Any 4 of the 8 rollouts in a K=8 group are what a K=4 group would have produced, so enumerating all C(8,4) subsets tells us, for each group, how often K=4 would have reported a wrong label. Two views of the same data follow:

  • Of the queries that are informative at K=8, about 75% would also be informative at K=4. The remaining 25% would be mislabelled as zero-variance by sampling noise alone.
  • Of the queries that K=4 would have labeled as zero-variance, a too-easy label is wrong 29.8% of the time, while a too-hard label is wrong only 16.9% of the time. Thus, recycling too-easy queries has more recoverable signal just from noise alone.

On a matched budget, N1 compares recycling and no-recycling at K=8. Raising K uniformly is costly, since every query pays for the larger group regardless of need. Recycling can be seen as an adaptive increase in K across policy updates (N2.1), so each query consumes only the rollouts it needs to yield a training signal.

W2, W3, Q2 - Ablations at the 4B parameter scale and multiple seeds

We agree these experiments would strengthen the empirical case. We have started the too-easy-only VS too-hard-only ablation at the 4B parameter scale. We will update both this thread and the global answer with any experiments that finish within the rebuttal window, and a revised version of the manuscript can include the results.

Thank you for the rebuttal. It addresses several of my concerns.

However, the K=8 gain is modest (+0.30), independent RL training seeds are still missing, and the 4B too-easy/too-hard ablation remains incomplete. The relative contributions of finite-K noise and policy changes also remain unclear.

Overall, the rebuttal strengthens the paper, but some empirical concerns remain. Currently, I maintain my original rating of 3.

Thanks for the follow-up, and for stating that the rebuttal solved several of your concerns.



The 4B too-hard-only arm has finished under a training setup that fully matches the paper:

Method2WikiBamboogleHotpotQAMuSiQueNQPopQATriviaQAAvg
4B recycle all79.184.074.441.374.455.889.971.3
4B too-hard only78.580.073.442.072.151.691.269.8

While unfortunatelly the too-easy-only arm is still running, current results are consistent with the ablation at the 1.4B scale, where recycling both too-easy and too-hard brings the strongest improvement.

As for the reviewer’s remaining points:

  • The K=8 experiment (N1, global answer) was conducted on a single training epoch to fit the rebuttal’s timeframe.
  • Regarding finite-K, we note that experiment N2.1 compares a setting where an arm only solves via finite-K noise mitigation (i.e., increasing K under the same policy), while the other only solves via policy change (i.e., deferring 4 rollouts to a future policy), and the latter achieved stronger performance.
Reviewer_ZSXmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

When GRPO with binary outcome rewards is used to train search agents, a group of rollouts that is all correct or all incorrect yields zero advantage, so the rollouts are wasted. To address this, the paper proposes query recycling: the training pool is a weighted set of queries; a query that produces a gradient signal has its weight set to zero and is removed from the pool, while a zero-variance query keeps its weight and can be sampled again in later steps. The authors argue that zero variance is a transient state that changes with the policy, not a fixed property of the query. The paper also unifies GRPO, DAPO, and the proposed method in a single framework defined by an oversampling factor k and a weight-update rule.

Experiments use synthetic data and a sandboxed retrieval environment, with all configurations compared under a matched budget of 84.5k rollouts. Qwen3-4B reaches an average Pass@1 of 71.3 over seven multi-hop QA benchmarks and Qwen3-1.7B reaches 66.0; the method is further validated on Llama3.2-3B and SmolLM3-3B. The analysis shows that, late in training, recycled queries account for about three quarters of the effective batch, with the "too easy" side contributing 40%.

Critical strengths

S1. Systematic structure and a complete line of argument. The paper starts from an observation: Figure 1 shows that queries flip between zero-variance and informative during training. It then identifies the problem: existing methods simply discard zero-variance queries and waste this signal. Next, it derives the motivation from the fact that the zero-variance probability depends on the policy, so zero variance is only a transient state. It then presents the method together with the unified k × weight-rule framework, and finally validates it with main results, ablations, and an analysis of training dynamics. Each step follows from the previous one, so the reader can see where the problem comes from and why the method is designed the way it is.

S2. A simple method with a concrete increment. The method changes only the sampling weights; it does not modify the reward function or add auxiliary models, so it can be plugged into existing GRPO/DAPO pipelines at very low cost. I could not find a precedent in published work for the specific rule "keep zero-variance queries, remove queries once they yield a signal." The ablations cover four settings: recycling only too-easy queries, recycling only too-hard queries, recycling all, and soft recycling.

S3. Broad experimental coverage and a useful analysis. The evaluation uses 10 benchmarks: 7 multi-hop QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle) plus 3 long-horizon benchmarks (GAIA, HLE, WebWalker). Experiments span 4 backbones (Qwen3-1.7B, Qwen3-4B, Llama3.2-3B, SmolLM3-3B), and the gains are consistent in direction under a matched budget. Several findings from the analysis hold even if the method itself is not optimal: zero-variance difficulty depends on the current policy; the "too easy" side accounts for 40% of the recycled signal; and the group size K jointly determines cost and the granularity of difficulty classification.

Critical weaknesses

W1. Incomplete baseline comparison. Table 1 already includes SmartSearch-3B and ORBIT-4B as baselines, which shows that 3B/4B models are within the scope of comparison. Yet from Table 3 of the BehaviorPrime paper (p.7), the authors take only the 1.7B row and omit the Llama3.2-3B row. The omitted row reports 82.0 on NQ (vs. 74.4 for the paper's 4B model) and a seven-benchmark average of 70.9, essentially on par with the paper's 4B result of 71.3 and 3.0 points higher than the paper's own Llama3.2-3B result of 67.9 on the same backbone. The abstract's claim of outperforming prior systems therefore does not hold under a same-backbone comparison. In addition, on 1.7B the main comparison baseline B-DAPO (64.1) is itself below GRPO (64.9); the reported +1.9 is the gain over B-DAPO, whereas the actual gain over GRPO is only +1.1.

W2. Overstated novelty and missing related work. The paper does not cite ERPO (ICLR 2026), which makes the same claim: it also treats zero-variance prompts as a source of signal that can be reactivated, differing only in that it raises the sampling temperature and covers only the all-correct side. The paper does not discuss the difference between the two at all. The paper states that prior methods "assume the zero-variance state persists," which contradicts the GRESO paper (NeurIPS 2025): GRESO explicitly reports that about 20% of zero-variance prompts become effective again and designs an adaptive exploration probability for this reason. Work in the same direction, such as PCL, ExGRPO (both ICLR 2026), and Prompt Replay, is also not cited. Several of these prioritize medium-difficulty prompts, the opposite of this paper's choice to retain both zero-signal extremes, and the paper does not argue why its choice is better.

Questions for authors

Q1 (Criteria for selecting baselines) What criteria were used to select the baseline methods and the specific result rows? Since Table 1 already includes 3B/4B models, why is BehaviorPrime-Llama3.2-3B (82.0 on NQ, 70.9 on average) omitted while the 1.7B row from the same table is included? Please add this row and revise the corresponding statement in the abstract; if you consider it not comparable, please explain the specific reasons.

1 comment

We thank the reviewer for the comments. Please find below our responses to each of the raised points.

W1/Q1 - BehaviorPrime LLaMA result

We thank the reviewer for pointing out the BehaviorPrime Llama3.2-3B result, which is included in the latest version of their paper (and not in the previous version that we analysed).

Our baseline selection focused on methods using Qwen models, i.e. the model family used in our main experiments. Nevertheless, we agree that the Llama3.2-3B result is relevant and that the abstract's broad claim of outperforming prior systems requires clarification. We will include this result and revise the corresponding claim in an updated version of the manuscript.

The comparison should also be contextualized, i.e. our training pipeline relies exclusively on synthetic queries and a simpler SFT filtering strategy, whereas BehaviorPrime uses benchmark-derived training splits (including NQ, where the reviewer noted the larger performance gap) and a more extensive mid-training strategy that filters trajectories to retain desirable search behaviors. These approaches are complementary to our RL method, which can be applied on top of either initialization.

W2 - Other related work

We thank the reviewer for bringing these papers to our attention. We will discuss them and clarify where needed:

  • ERPO raises the sampling temperature to force failures on K/K prompts, thus following a similar intuition from DAVID-GRPO, where the current policy is exploited to resolve the zero-variance issue, while we explore the usage of future policies. Regarding the too-easy-only setup, we ablate this under our setup on the main table of the paper.
  • We will clarify our positioning relative to GRESO. The authors do report that some zero-variance prompts become effective again, and handle them through an exploration probability. However, their design rests on persistence, and it uses this to skip prompts and save rollouts at matched accuracy. We take the opposite view, i.e., re-entering queries are a source of signal (Fig. 4), so we keep them at full sampling weight. The two methods also differ in their data requirements, as GRESO’s predictor needs per-prompt histories built over multiple epochs, which are not feasible in expensive domains such as search agents.
  • PCL, ExGRPO, Prompt Replay: These methods prioritize medium-difficulty prompts (pass rate = 0.5) to maximize variance. Recycling has a different target, i.e., recovering signal queries currently at 0/K or K/K. The approaches are orthogonal and can be combined, i.e., in a “variance-optimal” curriculum where previously zero-variance queries are only considered when their pass rate is approximately 50%.
Reviewer_QUVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author builds on the intuition of wasted zero-variance rollout group during GRPO for search agent training. By using a query recycling strategy, the proposed method retains the zero-variance queries for future resampling, while removing queries that have already contributed to gradient updates. Experiments across multiple backbones and multi-hop QA benchmarks demonstrate consistent improvements over GRPO and Bounded-DAPO under matched rollout budgets. Additional analyses examine recycling dynamics, group size, and alternative recycling strategies.

Critical strengths
  1. Clear motivation. Zero-variance groups indeed waste rollout computation, though somewhat quite well-understood property of GRPO. Reusing them as the policy evolves is reasonable.
  2. Consistent empirical improvements. Across four model configurations from three backbone families, recycling improves average performance over Bounded-DAPO by 0.9–1.9 percentage points under matched rollout budgets.
Critical weaknesses
  1. Somewhat limited methodological novelty. The central observation that zero-variance status can change as the policy evolves is relatively intuitive. The proposed method essentially retains previously uninformative queries for future resampling while removing those that have contributed to gradient updates. While practically useful, the algorithmic contribution appears incremental.
  2. Insufficient comparison with alternative sampling strategies. Although the paper investigates several recycling variants, the primary comparisons remain GRPO and Bounded-DAPO. It is unclear whether the gains arise specifically from the proposed recycling rule or more generally from alternative allocations of the sampling budget. Comparisons against random resampling or difficulty-aware sampling would strengthen the contribution.
Questions for authors
  1. The benefits of recycling are not fully disentangled. The proposed strategy simultaneously retains zero-variance queries and removes previously informative queries from the sampling pool. How much of the improvement comes from each decision?
  2. Furthermore, a query can transition from zero-variance to informative purely due to stochastic rollout sampling, even under a fixed policy. Could the authors quantify how much of the observed recycling benefit is attributable to policy evolution rather than stochastic resampling?
1 comment

We thank the reviewer for the comments. Please find below our responses to each of the raised points.

W1: Novelty

We agree that the method itself is simple. The contribution lies in three parts, namely the method itself, a synthetic-only environment for training small search agents, and the observation that zero-variance queries still carry training signal. We also analyze how the queries move in and out of zero-variance over the course of training. The global response contains new experiments that further solidify our claims, showing that the gains hold under larger groups, that they come from targeting zero-variance queries specifically rather than from re-sampling queries in general, and that recycling makes more efficient use of the rollout budget than uniformly increasing K to mitigate noise.

W2: Other sampling strategies

We have conducted two experiments which change the sampling strategy, namely through (a) another hyperparameter instantiation of our Soft Recycling generalization from Appendix C (global answer N2.1), and (b) through a random resampler (N2.2, a query is randomly returned to the pool irrespective of variance signal).

  • N2.1 shows results for a “recycle at most once” baseline, relevant to Finite-K sampling noise and higher-K scenario comparisons.
  • N2.2 shows that the improvements come from the variance signal, not from random replay.

Q1: Retaining zero-variance queries and removing previously informative queries... how much improvement from each?

Removing zero-variance queries is the standard approach in training, i.e., models are trained once on a datapoint, zero-variance datapoints produce no changes, and the datapoint is revisited only in subsequent epochs (e.g., the case for the GRPO and B-DAPO baselines). We build on top of this by allowing zero-variance queries to stay in the pool. In alignment with the author's question, Appendix C already extends this setting by letting informative queries return to the pool, although this did not improve the results. The random baseline (N2.2) also allows informative queries to return to the pool, and, consistently with the previous experiment, results did not improve (i.e., no result is superior to that of "recycle zero variance only").

Q2: Finite-K sampling noise

We agree that the signal that recycling recovers from zero-variance queries can come from two sources, namely policy change between steps or finite-K sampling noise. We also agree that the paper does not separate them. We will make this distinction explicit by extending the parameter K ablation in the main paper with the analysis from the N1 experiment (global response). Approximately 75% of queries that are informative at K=8 would also be informative at K=4 under the same policy. Thus, finite-K sampling noise causes K=4 to miss the signal from approximately a quarter of the queries that are informative at K=8. Raising K uniformly to remove this noise is costly, since every query pays for the larger group regardless of need. In fact, recycling can be interpreted as an adaptive increase of K across policy updates (N2.1), where each query consumes only the rollouts it needs to yield training signal.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Answer - Part 1

We thank the reviewers for their comments. We present new experiments in a global answer, followed by individual responses to each reviewer.

New Experiment 1 (N1) - Finite-K sampling noise

Reviewers QUVY and X3Zh had questions regarding the noise inherent to finite-K sampling on group-based RL algorithms. To address these issues, we further extend our K=8 experiments and analysis.

Setup:

  • Qwen3-1.7B, same SFT initialization, answer-only reward, oversampling k=2, B=32.
  • To fit the rebuttal period, every arm runs one epoch (165 steps), i.e. half the gradient steps of the main experiments. After the rebuttal, for an eventual camera-ready version, we can consider a setup matching the gradient steps of the main experiments.
  • Evaluation: 7 QA benchmarks, Pass@1 averaged over 8 inference passes.

Comparisons:

  • B-DAPO, K=8, without recycling.
  • B-DAPO, K=8, with recycling.
Method2WikiBamboogleHotpotQAMuSiQueNQPopQATriviaQAAvg
B-DAPO, K=866.8066.9065.2135.0871.6153.7888.0663.92
B-DAPO + recycling, K=868.2669.1066.5336.1169.8552.2587.4364.22

Analysis:

Assuming rollouts are iid given the policy, any 4 rollouts from a K=8 group form a valid K=4 sample. Enumerating all (C(8,4)) splits from our training logs indicates that approximately 75% of queries that produce an informative group at K=8 would also produce an informative group at K=4 under the same policy. Thus, finite-K misclassification affects a non-negligible fraction of queries that are informative at K=8, although most remain informative at K=4. Two observations follow:

  • The remaining 25% illustrates the finite-K misclassification that can occur with small GRPO groups, representing one source of inefficiency that our method aims to mitigate.
  • The gains from recycling persist at K=8, where misclassification is reduced. Even in this reduced setting, 48% of all sampled queries were zero-variance. The magnitude of the improvement is lower in this single-epoch scenario.
AuthorsOpenReview ↗

Global Answer - Part 2

New Experiment 2 (N2) - Other sampling strategies

N2.1 - Recycle at most once

In this experiment, we test a sampling variant of the soft recycling generalization from Appendix C. Informative queries are removed from the pool once trained on, and zero-variance queries can be recycled only once. This variant further addresses the finite-K concern raised by reviewers QUVY and X3Zh. Raising K aims to mitigate zero-variance by exploiting the current policy. Uniformly doing so reduces noise but is expensive, since queries may not need the extra rollouts. Recycling a zero-variance query once grants a second group to that query alone, so this variant adapts K per query.

Setup: Same as N1.

Comparisons:

  • B-DAPO, K=8, no recycling (from N1): every query always gets 8 rollouts.
  • B-DAPO, K=4, recycle at most once (soft recycling, weight 0, retire streak 2). Every query first gets 4 rollouts. If those are zero-variance, the query potentially gets 4 more at a later step, from the then-current policy (totaling 8), and is then retired. In other words, every query gets at most 8 rollouts, divided between policies.
MethodTotal rollouts2WikiBamboogleHotpotQAMuSiQueNQPopQATriviaQAAvg
B-DAPO, K=884,48066.8066.9065.2135.0871.6153.7888.0663.92
B-DAPO + recycle-once, K=442,24067.9070.2066.2836.5770.8354.4787.3564.80

Analysis:

  • Both arms take the same number of gradient steps on batches of informative groups. The difference is how many rollouts are spent to find those groups. As shown in N1, 75% of queries informative at K=8 are already informative with 4 rollouts. Thus, for most queries that yield an informative group at K=8, the additional 4 rollouts are not necessary to obtain training signal.
  • Recycle-once spends them only on queries whose first 4 rollouts were zero-variance, and spends them later, under the then-current policy.
  • I.e., the K=8 arm tries to solve zero-variance queries by exploiting the current policy, while the K=4 w/recycle arm defers to a future one.
  • Recycling can be interpreted as an adaptive allocation of rollout budget, i.e., queries initially classified as zero-variance receive additional rollouts under a later policy, whereas informative queries do not incur this additional cost.

N2.2 - Random pool management

Reviewer QUVY asked for a baseline that resamples queries at random. We executed this and tested whether recycling gains come from selecting zero-variance queries specifically, or from returning queries to the pool at all. At each step, a query has 50% chance to return to the pool, irrespective of variance signal.

Setup: Same as N1.

Comparisons:

  • B-DAPO, K=4, recycle at most once (from N2.1).
  • B-DAPO, K=4, random recycle (a query has 50% chance to return to the pool irrespective of variance signal).
Method2WikiBamboogleHotpotQAMuSiQueNQPopQATriviaQAAvg
B-DAPO + recycle-once, K=467.9070.2066.2836.5770.8354.4787.3564.80
B-DAPO + random, K=465.1667.8064.5835.0870.8351.7187.4363.23

Analysis:

Recycle-once spends all of its retries on zero-variance queries. Random spends 40.5% of them on queries that already yielded a training signal, and gives a second chance to only 26.8% of zero-variance queries (vs 53.6%). As a result, recycle-once trains on more distinct queries within the same rollout budget, and outperforms random by 1.57 points on average (64.80 vs 63.23).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.