acceptodds
Under review as a conference paper at ICLR 2027

Selecting What to Learn and Correcting Policy Gradients for Search Agents

Abstract

Search agents extend language models to information-seeking tasks that require repeated retrieval and reasoning across sources. Learning from rewards based on final-answer correctness remains difficult. Questions that are too easy or too hard provide little reward contrast, while sampled trajectories produce noisy gradients. We introduce an adaptive training framework that couples question selection with gradient correction. A frozen generator constructs questions from retrieved evidence, and independent probes estimate the current agent's success rate on each question. The framework then routes questions to supervised fine-tuning, reinforcement learning, or later reassessment. For reinforcement learning, we propose Gradient-Corrected Group Relative Policy Optimization (GC-GRPO) to reduce sampling noise in policy gradients. GC-GRPO constructs zero-mean reference gradients from current trajectories and combines them using weights learned from earlier batches. Our analysis shows when this correction reduces mean squared estimation error while preserving the expected GRPO gradient at the sampling policy. Across five search benchmarks, the trained 4B search agent achieves 85.44% accuracy on GAIA, 48.66% on BrowseComp, 47.75% on BrowseComp-ZH, 47.75% on SEAL-0, and 82.00% on XBench-DS.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agentic search, self-improvement, reinforcement learning, curriculum learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_XdAjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses two challenges in training search agents with reinforcement learning: selecting informative training questions and reducing noise in policy gradients. The authors propose an adaptive training framework that generates evidence-grounded questions and routes them to SFT, RL, or later reassessment based on the agent's current success rate. They further introduce GC-GRPO, which uses zero-mean reference gradients constructed from token-prefix features and peer-trajectory rewards to reduce gradient estimation error while preserving the expected GRPO gradient at the sampling policy.

Critical strengths
  1. GC-GRPO provides an interesting application of control variates to group-relative policy optimization. I particularly appreciate that the authors go beyond benchmark performance and directly measure gradient estimation error in Figure 4. This provides useful evidence for the proposed mechanism.
  2. Apt consistently improves over its initial model across five benchmarks, with competitive performance against existing search agents. The ablations in Tables 3 and 4 also provide evidence that both adaptive question selection and gradient correction contribute to the improvements.
Critical weaknesses
  1. The training-cost comparison is not fully controlled. In Section 4.3, the adaptive framework is compared against direct RL using fewer RL steps. However, the adaptive framework additionally requires multiple search rollouts for question probing and SFT training. These costs are not accounted for in the comparison. Consequently, it is unclear whether the adaptive framework is more efficient under the same overall compute or trajectory budget. A compute-matched comparison would make the claimed efficiency benefits more convincing.
  2. Limited evidence for the robustness and generalizability of the improvements. The experiments are conducted with a single 4B backbone, and no results across multiple training seeds are reported. Some improvements over competing RL algorithms are relatively small, particularly given that several evaluation benchmarks contain only around 100 questions. Moreover, the search system uses a separate 27B model for evidence extraction, making comparisons with externally reported search-agent results difficult to interpret without matched tool configurations.
Questions for authors
  1. Could the authors provide a comparison under matched overall training costs, including question probing, SFT, and RL? It would also be useful to compare against random or fixed question routing with the same SFT/RL allocation to better isolate the benefit of adaptive selection.
  2. How stable are the reported benchmark improvements across different training seeds? Could the authors also clarify the contribution of the 27B evidence extraction service to the reported performance, particularly when comparing Apt with other 4B search agents?
7 comments
AuthorsOpenReview ↗

Comment 1. Training time including probing, SFT, and RL

We thank the reviewer for recognizing adaptive question selection, GC-GRPO, and the direct gradient diagnostic. Below, we provide complete training-node timing, a comparison within a common time budget, a random-routing control, independent training repeats, and direct tests of GC-GRPO and the page-summary service. Changes in the revised manuscript are highlighted in yellow.


The two settings in Section 4.3 share the initialization, candidate questions, hardware, and RL implementation, and differ in how questions are used for training. Their configurations are summarized below.

\hline \text{Configuration} & \text{Direct RL} & \text{Adaptive framework} \\\\ \hline \text{Initialization} & \text{Qwen3.5-4B} & \text{Qwen3.5-4B} \\\\ \text{Candidate questions} & 1{,}024 & 1{,}024 \\\\ \text{Training node} & \text{8 H200 GPUs} & \text{8 H200 GPUs} \\\\ \text{Training framework} & \text{Slime} & \text{Slime} \\\\ \text{RL algorithm} & \text{GC-GRPO} & \text{GC-GRPO} \\\\ \text{RL batch} & \text{8 questions}\times\text{8 rollouts} & \text{8 questions}\times\text{8 rollouts} \\\\ \text{RL rollout/update GPUs} & 4/4 & 4/4 \\\\ \text{Screening/probing} & \text{No} & \text{Yes} \\\\ \text{SFT questions} & 0 & 353 \\\\ \text{RL questions} & 1{,}024 & 619 \\\\ \text{RL updates} & 128 & 82 \\\\ \hline \end{array}$$ Both RL stages use four GPUs for SGLang rollout and four for Megatron updates, with asynchronous overlap. Adaptive training uses all eight GPUs during probing and SFT. Of the 1,024 candidates, 31 are filtered, 353 supply successful SFT demonstrations, 619 enter RL, and 21 remain for reassessment. The 619 RL questions yield 82 updates when batched by eight within each routing batch. We extracted elapsed training-node times from the training logs, covering screening/probing, SFT, and RL sampling and updates. $$\begin{array}{lrr} \hline \text{Stage (elapsed hours)} & \text{Direct RL} & \text{Adaptive framework} \\\\ \hline \text{Screen/probe/reassess} & 0 & 26.9 \\\\ \text{SFT} & 0 & 8.3 \\\\ \text{RL sampling + updates} & 48.3 & 21.8 \\\\ \text{Total} & 48.3 & 57.0 \\\\ \hline \end{array}$$ Adaptive training reduces RL updates by 35.9% and RL-stage time by 54.9%; including probing and SFT, its completed run takes 18.0% longer overall. To compare performance at comparable training times, we use Direct RL's completion time of 48.3 hours as the reference and evaluate the nearest available Adaptive checkpoint within that budget. This checkpoint was saved at 44.0 hours, after 54 RL updates. The completed Adaptive model at 57.0 hours is also shown. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Direct (48.3 h)} & \text{Adaptive (44.0 h)} & \text{Adaptive (57.0 h)} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 78.64 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 31.28 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 33.22 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 34.23 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 75.00 & 77.00 \\\\ \hline \end{array}$$ The 44.0-hour checkpoint is higher on three benchmarks and lower on two than Direct RL, with a 4.00-point gain on XBench-DS. Completing the adaptive schedule improves four of five benchmarks over Direct RL.
AuthorsOpenReview ↗

Comment 2. Adaptive selection versus random question routing

Success-based routing outperforms random routing on all five benchmarks by 7.76 to 10.28 percentage points. The control retains the Section 4.3 candidate pool, initialization, GC-GRPO, and training and evaluation settings. It randomizes which questions enter SFT or RL while keeping the numbers of questions assigned to each stage the same. The same probing procedure supplies successful demonstrations for SFT, but its success-count intervals no longer determine the assignment. SFT requires a successful demonstration; questions with no observed success may enter RL but cannot enter SFT.

\hline \text{Benchmark} & \text{Direct RL} & \text{Random routing} & \text{Adaptive routing} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 72.82 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 25.99 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 26.30 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 24.32 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 68.00 & 77.00 \\\\ \hline \end{array}$$ Section 2.2 explains why objective assignment matters. All-success or all-failure rollout groups have zero group-relative task-reward advantages. For eight independent binary-reward rollouts with success probability $p$, the probability of reward contrast is $1-p^8-(1-p)^8$, maximized at $p=1/2$. Random allocation can therefore spend RL sampling on questions less likely to provide reward contrast, while passing up successful demonstrations that could provide direct SFT supervision. This issue also motivates DAPO's dynamic sampling and query recycling for agentic search. The results support matching questions to learning objectives beyond simply combining SFT and RL. Random routing is now included in Table 2, and Section 2.2 of the revised manuscript cites DAPO alongside query recycling.
AuthorsOpenReview ↗

Comment 3. Stability across training seeds

Following your suggestion, we used our available training resources to complete two additional independent seed runs under the Section 4.3 settings. This experiment uses our complete adaptive framework with GC-GRPO on a smaller pool of 1,024 training questions. Apt in the main comparison is trained with 6,400 questions; the routing, SFT, and GC-GRPO components are retained in these repetitions.

Seed 1 denotes the Table 2 Adaptive run; Seeds 2 and 3 are the additional independent training runs, with the initialization, candidate pool, training settings, and tools held the same. Mean and sample SD are computed across the three seeds.

\hline \text{Benchmark} & \text{Seed 1} & \text{Seed 2} & \text{Seed 3} & \text{Mean} & \text{SD} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{pp} \\\\ \hline \text{GAIA} & 80.58 & 79.61 & 81.55 & 80.58 & 0.97 \\\\ \text{BrowseComp} & 36.27 & 36.97 & 35.31 & 36.18 & 0.83 \\\\ \text{BrowseComp-ZH} & 35.99 & 37.02 & 34.95 & 35.99 & 1.04 \\\\ \text{SEAL-0} & 33.33 & 33.33 & 31.53 & 32.73 & 1.04 \\\\ \text{XBench-DS} & 77.00 & 81.00 & 79.00 & 79.00 & 2.00 \\\\ \hline \end{array}$$ All three runs exceed Direct RL on GAIA, BrowseComp, BrowseComp-ZH, and XBench-DS, and remain lower on SEAL-0. **The direction of the framework's benchmark gains is reproduced across seeds, with SDs of 0.83 to 1.04 points on four benchmarks and 2.00 points on XBench-DS.** The Section 4.3 Adaptive run takes 57.0 hours on the eight-H200 node, while full Apt training takes less than a week. These repetitions assess the same proposed method at a smaller training-data scale. Given our limited training resources, we prioritized the two additional seed runs and appreciate your understanding that we could not train additional backbones within the discussion period.
Reviewer_A6auOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attacks two challenges in training LLM-based search agents: first, some questions give little useful feedback to the agent, and second, randomly sampled rollouts to the questions produce noisy gradient updates.

The paper has two contributions. First, it proposes a training curriculum where it first records the agent's success rate: questions with rare successful attempts get routed to SFT training, while questions with moderate success rate get routed to RL training. Second, it proposes gradient-corrected GRPO (GC-GRPO), where it adds a learned correction to GRPO's gradient estimate to reduce sampling variance (technically, it's a token-level policy-gradient baseline, with 10 coefficients fitted from earlier update steps). The authors prove conditions under which the correction preserves the expected GRPO gradient while reducing variance. Finally, they show that the resulting 4B agent beats or nearly matches SOTA on five benchmarks (including GAIA, BrowseComp, etc.).

Critical strengths

Performance and ablation analysis of the system. This paper empirically investigates the effectiveness of the individual components of the combined system, demonstrated to work well across five search-agent benchmarks (instead of just reporting on the performance of the whole system). The adaptive SFT+RL curriculum does outperform RL alone on four out of five benchmarks, and GC-GRPO outperforms vanilla GRPO on all five benchmarks (within the curriculum framework). This gives some confidence on the modular usefulness of the proposed pipeline beyond this specific search-agent setting.

GC-GRPO's gradient correction analysis. GC-GRPO is the more algorithmically interesting contribution in this work; I like that the authors build this baseline for the policy gradient carefully, using historical coefficients, leave-one-out peer rewards, etc.

Moreover, I like the neat diagnostics used for Figure 4 for estimating the improvements in estimating the true gradient under MSE in real training runs (which just needs taking norm differences between the vanilla GRPO gradient vs the corrected gradient). And indeed, Figure 4 shows improvements in reducing sampling noise in the runs. The diagnostic has some limitations though, which as I understand, only shows absolute reduction in noise, but not the proportion of noise reduced.

Critical weaknesses

My biggest concern is whether the particular ten-feature design of the baseline in GC-GRPO is really necessary. While the experiments show that the correction does reduce sampling noise, there is little provided evidence for why a much simpler baseline (e.g. just one learned coefficient) would not work comparably well. I think the complexity of the baseline function is part of this algorithm’s design, and deserves further analysis. For instance, following from the argument in 3.1, a priori, we cannot just assume that a basic one-feature baseline (say fi=c0f_i = c_0) or with an added peer-success term fi=c0+c1(Rˉ−i−12)f_i = c_0 + c_1(\bar{R}_{-i} - \frac{1}{2}) would not work comparably well and how much added benefit the 10-feature baseline has.

Questions for authors
  1. Following from my weakness critique, could the authors give a more thorough analysis that help us better understand the feature-complexity of the baseline, beyond the ten-feature one used in the paper?

  2. In table 2 the full method (adaptive system with GC-GRPO) gives 36.27% accuracy but in Table 4 it scores 48.66%. Could the authors clarify the configurations across the tables?

  3. Some published works related to GC-GRPO should be cited [1-2] -- they share the design philosophy of building efficient token-level baseline functions to reduce sampling noise/variance and are focused on LLM (long-horizon) RL.

References.

  1. Li Y., Xu J., Li Z., Liu J., Liu W., Tong Y., Zheng L., Xue Z., Zhang Y., Cai T., et al. The optimal token baseline: Variance reduction for long-horizon llm-rl. arXiv preprint arXiv:2602.07078, 2026.

  2. Zeng G., Zhou Z., Arora D., and Zanette A. Shrinking the variance: Shrinkage baselines for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2511.03710, 2025.

6 comments
AuthorsOpenReview ↗

Comment 1. Does the full feature family outperform simpler baselines?

We thank the reviewer for recognizing the component studies and gradient analysis. Changes in the revised manuscript are highlighted in yellow.


The ten-feature correction outperforms the one- and two-feature baselines suggested by the reviewer on all five benchmarks. Here, peer success means the mean reward Rˉ−i\bar R_{-i} of the other G−1G-1 trajectories for the same question, excluding trajectory ii itself. We use its centered value ri=Rˉ−i−12r_i=\bar R_{-i}-\tfrac12.

Constant is the reviewer's proposed fi=c0f_i=c_0, and Constant + peer is fi=c0+c1rif_i=c_0+c_1r_i. Prefix and Full are the variants named in Table 4. Prefix uses the five features (1,uit,uit2,pend,it,Hit)(1,u_{it},u_{it}^{2},p_{\mathrm{end},it},H_{it}), describing a constant, generation progress, stopping tendency, and uncertainty. Full adds their five products with rir_i, allowing peer success to modulate the correction. KK below counts these features. All rows retain adaptive probing and SFT, and corrected variants fit their coefficients from preceding batches.

\hline \text{Family} & K & \text{GAIA} & \text{BrowseComp} & \text{BrowseComp-ZH} & \text{SEAL-0} & \text{XBench-DS} \\\\ & & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GRPO} & 0 & 77.67 & 45.73 & 35.29 & 35.14 & 79.00 \\\\ \text{Constant} & 1 & 79.61 & 46.68 & 36.33 & 37.84 & 79.00 \\\\ \text{Constant + peer} & 2 & 80.58 & 47.08 & 36.33 & 36.94 & 79.00 \\\\ \text{Prefix} & 5 & 82.52 & 47.95 & 46.02 & 45.95 & 81.00 \\\\ \text{Full} & 10 & \mathbf{85.44} & \mathbf{48.66} & \mathbf{47.75} & \mathbf{47.75} & \mathbf{82.00} \\\\ \hline \end{array}$$ The simpler baselines recover part of the improvement over GRPO, but do not match Full. Relative to $K=2$, Full improves BrowseComp-ZH by 11.42 percentage points and SEAL-0 by 10.81 points. Full also exceeds the prefix-only $K=5$ variant on all five benchmarks by 0.71 to 2.92 points. Thus, the observed gains come from conditioning the correction on generation state and its interactions with peer success, beyond learning one or two coefficients.
AuthorsOpenReview ↗

Comment 2. Why do Tables 2 and 4 report different BrowseComp scores?

The 36.27% and 48.66% scores use the same complete method and training hyperparameters, with different amounts of training data. All Section 4.3 experiments use the fixed pool of 1,024 candidate questions, producing Table 2 and Figure 3. The full-method result in Table 4 uses all 6,400 questions. In the original Apt training, the Figure 1 framework generates 128 new questions each round, probes them together with retained questions, performs SFT, and then samples fresh RL trajectories. Apt completed 50 rounds. The iterative ablations reuse these 6,400 questions in the same order and 128-question round divisions, keeping the training and evaluation settings unchanged.

We acknowledge a presentation oversight in Table 4's second result row, Direct RL. It reuses Table 2's first-column results, but we did not distinguish its 1,024-question scope. We have now labeled and separated this row as a fixed-pool reference in the revised Table 4 and specified the training-data scopes in its caption. All other trained Table 4 variants use the full 6,400-question setting. The additional K=1K=1 and K=2K=2 experiments in Comment 1 follow this same full-data setting.

AuthorsOpenReview ↗

Comment 3. Effect size of the gradient diagnostic

Thank you for recognizing the gradient diagnostic and suggesting a clearer measure of its effect size. We have added a per-round quantitative analysis, with reductions equal to 8.54% to 13.38% of the GRPO gradient second moment. This uses the same paired batches as Figure 4, with policy, reference model, and correction coefficients fixed within each state.

Let As=∣Xs∣2‾A_s=\overline{\\|X_s\\|^2}, Bs=∣Ys∣2‾B_s=\overline{\\|Y_s\\|^2}, and Δs=As−Bs\Delta_s=A_s-B_s, where XsX_s and YsY_s are the original and corrected gradient estimates. We report

qs(q_s(\\%)=100\frac{\Delta_s}{A_s}.

The denominator is the gradient second moment, which includes the mean-gradient signal and sampling variability; it is not an estimate of centered sampling variance. Below, each column is a round and nsn_s is its number of independent complete batches. Displayed quantities are rounded independently; percentages use full-precision values.

\hline s & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\\\ \hline n_s & 8 & 16 & 16 & 16 & 16 & 16 & 16 & 16 & 16 & 16 \\\\ A_s & 3.2408 & 3.2328 & 3.1839 & 2.8968 & 2.9524 & 3.0105 & 3.1108 & 2.9461 & 2.8378 & 2.9047 \\\\ B_s & 2.9569 & 2.8983 & 2.9011 & 2.5809 & 2.6591 & 2.7535 & 2.6946 & 2.6763 & 2.5552 & 2.5357 \\\\ \Delta_s & 0.2839 & 0.3345 & 0.2828 & 0.3160 & 0.2934 & 0.2571 & 0.4162 & 0.2698 & 0.2826 & 0.3689 \\\\ q_s\\,(\\%) & 8.76 & 10.35 & 8.88 & 10.91 & 9.94 & 8.54 & 13.38 & 9.16 & 9.96 & 12.70 \\\\ \hline \end{array}$$ $$\begin{array}{lrrrrrrrrrr} \hline s & 11 & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 \\\\ \hline n_s & 16 & 16 & 16 & 16 & 16 & 16 & 16 & 16 & 16 & 16 \\\\ A_s & 2.8190 & 2.8038 & 2.6970 & 2.7183 & 2.6613 & 2.4705 & 2.4687 & 2.4243 & 2.4012 & 2.1617 \\\\ B_s & 2.5251 & 2.5585 & 2.4263 & 2.4482 & 2.3881 & 2.2027 & 2.2343 & 2.1860 & 2.1200 & 1.8948 \\\\ \Delta_s & 0.2939 & 0.2453 & 0.2707 & 0.2701 & 0.2732 & 0.2677 & 0.2344 & 0.2383 & 0.2812 & 0.2668 \\\\ q_s\\,(\\%) & 10.43 & 8.75 & 10.04 & 9.94 & 10.27 & 10.84 & 9.49 & 9.83 & 11.71 & 12.34 \\\\ \hline \end{array}$$ We have added these per-round percentages in a separately labeled strip below the round axis in the updated Figure 4, so that the original absolute reductions and their normalized magnitudes can be read together.
Reviewer_FaJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies search agents, and introduces and adaptive training framework for training search agents. Their framework has two main components:

  1. an Adaptive routing algorithm that decides in each round if a query should be used for RL training, as an SFT example, or not used in training. In each round, a frozen DeepSeek-v4-flash generates new 128 questions it creates from wikipedia articles. For each query, based on the current policy, the routing algorithm generates 8 rollout attempts for each query and the ones that get solved 0, times get saved for later, the ones that get correct answers 1,2,7, or 8 times is used as an SFT training example and the ones for 3,4,5,6 are use for RL post training.
  2. For RL post training, they introduce GC-GRPO - Gradient Corrected-Group Relative Policy Optimization, that aims to reduce the estimation noise in the GRPO gradient. GC-GRPO estimates 10 more reference gradients and their weighted sum is used as the gradient correction. The 10 reference gradients' weights are learned are based of 10 statistics that represent the current context and exponential averaging from previous batches. In each round, after the SFT is done on the query that got 1,2,7,8 correct rollouts, the ones with 3,4,5,6 correct rollouts originally are used for RL post training using the GC-GRPO algorithm where the reward is binary if the answer was accepted or if the correct doc was retrieved.

The authors train a 4B model initialized from Qwen3.5-4B called Apt based on this adaptive training framework. The authors evaluate Apt on 5 benchmarks and get the best performance on two and second best on 3 compared to search agent baselines.

Critical strengths

1, GC-GRPO is a novel noise reduction method for GRPO with evidence that it estimates the gradients with a smaller MSE compared to plain GRPO. Using the statistics that depend on the current context but are independent from the current query is a good way to obtain independent gradients with 0 expected mean. 2. The idea of using a router to route queries to SFT and RL in each round can be an interesting approach to utilize the available training given we are able to do it efficiently. 3. The results reported are strong and the resulting model achieves top performance in 2 and second best in 3 benchmarks compared to other 4B agents. 4. There are ablation experiments attempting to quantify the contributions of the different components.

Critical weaknesses

My main concerns with this paper is that the experiments have confounded effects or undisclosed/unequal training budgets that make some of the stated results less confident.

  • Comparison of adaptive training vs RL only is not fair : Section 4.3 compares RL only vs Adaptive Training that includes Routing + SFT + RL. Figure 3, compare training reward, but since RL only gets examples which are correct only 0,1,2 7,8 times, the training reward distribution can be very different as RL only has harder examples as well. Second, even if we assume rewards are comparable, the x axis is RL training steps, not any actual measure of compute. In on policy RL training, the rollout generation is the major bottleneck and the adaptive training framework requires rollout generation for all examples in the routing phase as well. This means that RL only did 12888 ~8k rollouts whereas the adaptive routing had to do 10248 + 828*8 ~13K rollouts. So it's not clear that the extra compute spend on routing + SFT could if dedicated to doing RL only with more rollouts could not achieve the same performance.
  • Effect of adaptively selecting RL candidates by routing is not reliably singled out : Table 2 is supposed to be trained on1024 Q and apparently not the same setup at apt final metrics. This is evident as the "adaptive framework" in Table 2 get 80.58/36.27/35.99/33.33/77.00 metrics, and Table 4 apt row get's 85.44/48.66/47.75/47.75/82.00. But to measure the effective of data probing in Table 4, the RL only row uses the same numbers are Table 2, but the RL + Adaptive probing get's 80.58/48.50/44.64/45.95/80.00 which is much more than the full adaptive framework number reported in table 2. This probably means that the training budget for the RL only and RL + Probing rows in Table 4 is not the comparable, so statements like "robing then raises these scores by 16.99 and 11.77 points" are not well supported.
  • Effect of GC-GRPO compared to other RL algorithms is not singled out: Table 3 compares GC-GRPO with GRPO, DrGRPO and GSPO but this comparison is only done within the adaptive training framework. Since the examples SFT and RL is performed on in subsequent rounds depends on the model performance itself, it means that the examples used in RL training in the later rounds depends on the RL algorithm itself. So even though table 3 shows GC-GRPO works well in the adaptive setting, it does not imply that the performance of GC-GRPO will translate to other settings. Since GC-GRPO is an independent RL algorithm, it could have been compared in isolation with other algorithms in a RL only setup.
  • Comparison of Apt with other 4B agents is not fair: The numbers for other agents are taken from their papers. Apt's evaluation used a Qwen3.8-27B in the evaluation environment that "extracts page evidence for the requested goal". So it's not fair to call it a 4B agent system, as other search agents compared in the analysis might not have access to a 27B model to extract page evidence. Also Apt is trained on wikipedia questions created by a stronger deepseek model. There is no commentary on whether that has any intersections with the evaluation datasets.
Questions for authors
  • Do Table 4’s RL-only and RL + probing rows use the same training budget? For rows of Tables 2, 3 and 4, can you please report number of questions, rounds, total rollouts, and gpu hours?
  • Can you plot Fig. 3 or equivalent against total sampled trajectories or GPU-hours and measure reward only on the common set of queries?
  • Does the adaptive framework still beat RL only at an equal rollout budget? Is it possible to run a version of experimentation in section 4.3 with similar rollout budget?
  • Can you compare GC-GRPO, GRPO, Dr. GRPO and GSPO using RL only runs? Ideally these should be repeated runs to report the std in estimates as well.
  • What happens if we replace the 27B model used to extract the page evidence with the model itself or with a separate 4B agent instead of a 27B agent?
7 comments
AuthorsOpenReview ↗

Comment 1. The shared question pool, Figure 3, and total training cost

We thank the reviewer for recognizing the gradient-correction method, its direct diagnostic, and Apt's performance. We clarify the shared inputs and training scopes, provide training-node timing and a comparison within the Direct RL time budget, and report targeted random-routing, RL-only, and page-summary substitution controls. Changes in the revised manuscript are highlighted in yellow.


Direct RL uses all 1,024 candidate questions, not only those with success counts of 0, 1, 2, 7, or 8. Section 4.3 states that “direct RL trains on all pairs.” Success-based routing applies to the Adaptive arm. With KK denoting successes in eight probes, K=0K=0 is deferred, successful trajectories from K∈1,2,7,8K\in\\{1,2,7,8\\} supply SFT, and questions with K∈3,4,5,6K\in\\{3,4,5,6\\} receive fresh RL rollouts. Both settings share the candidate pool, initialization, and GC-GRPO. Figure 3 records training reward; Table 2 evaluates final performance on common benchmarks.

Of the 1,024 candidates, 31 are filtered, 353 supply successful SFT demonstrations, 619 enter RL, and 21 remain for reassessment. Batching the RL questions by eight within each routing batch gives 82 updates, compared with 128 for Direct RL.

We also extracted training-node times for probing, SFT, and RL to account for the additional stages. Both runs use an eight-H200 node and Slime. RL uses four GPUs for SGLang rollout and four for Megatron updates with asynchronous overlap; Adaptive uses all eight GPUs for probing and SFT before RL.

\hline \text{Training-node quantity} & \text{Direct RL} & \text{Adaptive} \\\\ \hline \text{Screen/probe/reassess (h)} & 0 & 26.9 \\\\ \text{SFT (h)} & 0 & 8.3 \\\\ \text{RL sampling + updates (h)} & 48.3 & 21.8 \\\\ \text{Total elapsed time (h)} & 48.3 & 57.0 \\\\ \text{Rollout output (M tokens)} & 24.8 & 42.3 \\\\ \hline \end{array}$$ Routing reduces RL updates by 35.9% and RL-stage time by 54.9%; including probing and SFT, the completed Adaptive run takes 18.0% longer overall. To compare performance within a common training-time budget, we use Direct RL's completion at 48.3 hours as the reference and evaluate the nearest available Adaptive checkpoint within that budget. It was saved at 44.0 hours after 54 RL updates. The completed Adaptive model is also shown. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Direct (48.3 h)} & \text{Adaptive (44.0 h)} & \text{Adaptive (57.0 h)} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 78.64 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 31.28 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 33.22 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 34.23 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 75.00 & 77.00 \\\\ \hline \end{array}$$ The 44.0-hour checkpoint is higher on three benchmarks and lower on two, including a 4.00-point gain on XBench-DS. Completing the adaptive schedule improves four of five benchmarks over Direct RL. The common candidate pool controls the input questions, while routing changes which questions enter RL. Figure 3 therefore describes training dynamics on the respective RL pools. The added checkpoint comparison addresses training time; an exactly matched rollout-budget run and a common-query reward curve against cumulative resources have not been completed. We will clarify the figure's scope and report the stage costs in Section 4.3.
AuthorsOpenReview ↗

Comment 2. Training scope and the interpretation of Table 4

Table 4's Direct RL and RL + probing rows do not use the same training budget. Direct RL reuses Table 2's 1,024-question, 128-update run; RL + probing uses the full 6,400-question, 50-round schedule. We acknowledge that the original presentation did not distinguish this reference clearly. The revised Table 4 now separates it as a fixed-pool reference and specifies the training-data scopes in its caption. Section 4.5 will no longer attribute the cross-scope differences of 16.99 and 11.77 points solely to probing.

All Table 2 settings share exactly the same 1,024 candidate questions. All Table 3 algorithms and the other trained Table 4 variants share exactly the same 6,400 candidate questions and introduction order. The reused Direct RL reference is the exception in Table 4. The scopes are summarized below.

\hline \text{Setting} & \text{Candidate questions} & \text{Schedule} \\\\ \hline \text{Table 2, Direct RL} & 1{,}024 & \text{128 RL updates} \\\\ \text{Table 2, Adaptive} & 1{,}024 & \text{10 routing batches, 82 RL updates} \\\\ \text{Table 3, all algorithms} & 6{,}400 & \text{50 rounds, 128 new candidates per round} \\\\ \text{Table 4, Direct reference} & 1{,}024 & \text{Reused Table 2 Direct RL run} \\\\ \text{Table 4, other trained variants} & 6{,}400 & \text{50 rounds, 128 new candidates per round} \\\\ \hline \end{array}$$ Apt follows Figure 1, generating 128 new questions per round, probing them together with retained questions, performing SFT, and then sampling fresh RL trajectories. Other iterative configurations replay the resulting questions in the same 50 round divisions, with the same routing rule. Actual assignments respond to each policy's current capabilities. The 36.27% and 48.66% full-method scores therefore use the same method and training hyperparameters at different training-data scales. The completed Apt run took 158.7 training-node hours in the training logs. Long runs on the cluster are affected by system-load variation, so this is the observed time for that Apt run, not a runtime assigned to all 6,400-question variants. Section 4.3's fixed-pool stage costs are reported in Comment 1. **To isolate the assignment rule, we added a random-routing control under the Section 4.3 settings.** It retains the same initialization, 1,024 questions, GC-GRPO, and evaluation protocol, randomizing question identities while keeping the numbers assigned to SFT and RL the same. Probing still supplies successful SFT demonstrations, but success-count intervals no longer determine assignment. Questions with no observed success may enter RL but cannot enter SFT. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Direct RL} & \text{Random routing} & \text{Adaptive routing} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 72.82 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 25.99 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 26.30 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 24.32 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 68.00 & 77.00 \\\\ \hline \end{array}$$ Adaptive routing exceeds Random on all five benchmarks by 7.76 to 10.28 percentage points. This supports success-based assignment beyond simply combining SFT and RL, consistent with Section 2.2's rationale of directing questions likely to provide reward contrast to RL. We will add this control to Table 2.
AuthorsOpenReview ↗

Comment 3. GC-GRPO without adaptive routing or SFT

The additional RL-only comparison shows GC-GRPO improving over GRPO on all five benchmarks. We compare the two algorithms on the same fixed 1,024-question pool with probing, routing, and SFT disabled, using the same initialization and evaluation protocol. The GC-GRPO column corresponds to the Direct RL configuration in Section 4.3.

\hline \text{Benchmark} & \text{RL-only GRPO} & \text{RL-only GC-GRPO} & \text{Gain} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{pp} \\\\ \hline \text{GAIA} & 76.70 & 77.67 & +0.97 \\\\ \text{BrowseComp} & 29.70 & 31.51 & +1.81 \\\\ \text{BrowseComp-ZH} & 29.41 & 32.87 & +3.46 \\\\ \text{SEAL-0} & 28.83 & 36.03 & +7.20 \\\\ \text{XBench-DS} & 68.00 & 71.00 & +3.00 \\\\ \hline \end{array}$$ The gains range from 0.97 to 7.20 percentage points. This provides downstream evidence that GC-GRPO's benefit can also occur without adaptive routing or SFT. Table 3 tests algorithm choice within the framework. GRPO, GSPO, Dr. GRPO, and GC-GRPO receive the same 6,400 candidate questions in the same introduction order, while the shared routing rule responds to each learned policy. This evaluates the algorithms in the intended adaptive loop and demonstrates that the framework accommodates different RL update rules. The new RL-only control separately tests the correction without that loop. Figure 4 directly tests the gradient mechanism. At each frozen state, the policy, reference model, correction coefficients, and questions are fixed; GRPO and GC-GRPO gradients are computed on the same trajectory batches. Mean gradient-MSE reduction is positive at all 20 states, with each state's 95% interval above zero. Given the multi-day resource requirements of long-horizon RL, we prioritized the direct GRPO/GC-GRPO control and appreciate your understanding.
Reviewer_rM62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a training framework for search agents that combines adaptive use of generated questions with a gradient-correction method, GC-GRPO. A frozen generator constructs evidence-grounded questions, and the agent's success on independent probe rollouts determines whether a question is used for supervised fine-tuning, reinforcement learning, or later reassessment. GC-GRPO augments the GRPO gradient with zero-mean control variates derived from token-prefix features and other rollouts' outcomes, using coefficients estimated from earlier batches. The resulting 4B agent, Apt, is evaluated on five search benchmarks. The paper's two main claims are that success-based routing makes training data more useful and that the gradient correction improves optimization without changing the expected GRPO gradient at the sampling policy.

Critical strengths
  1. A carefully scoped variance-reduction construction. Section 3 and Appendices A–B distinguish preservation of the implemented GRPO gradient's expectation from unbiased estimation of the underlying task gradient. Fitting coefficients on earlier batches is important for the zero-mean argument, and the scalar correction can be incorporated into the existing backward pass. This is a concrete, technically motivated modification rather than an additional reward heuristic.

  2. Direct evidence for the proposed optimization mechanism. Figure 4 compares gradient estimates on paired trajectory batches at 20 training states. The reported confidence intervals for mean gradient-estimation MSE reduction are above zero at every state. This diagnostic is a particular strength: it tests whether the correction reduces sampling error, rather than inferring that mechanism solely from downstream accuracy.

  3. Strong reported system performance and useful component studies. Tables 1 and 4 show substantial improvements over the initial Qwen3.5-4B model on all five benchmarks. The paper also compares several policy-optimization algorithms under the adaptive framework and evaluates BrowseComp on its full 1,266-question set. Together, these results make the combined system worth investigating, even though the source and robustness of individual gains require further clarification.

Critical weaknesses
  1. The training-budget information is insufficient to interpret the component gains. Table 4's direct-RL row reproduces the 1,024-question, 128-update experiment in Table 2, whereas the full configuration reproduces Apt's main results. The realized budgets of the intermediate rows are not specified, so it is unclear whether the large differences isolate components or also reflect different training exposure. Furthermore, Figure 3 compares rewards on differently selected question pools, and its step count excludes probing and SFT costs. The existing results therefore suggest benefits from adaptation, but do not establish the claimed efficiency under a matched total budget.

  2. The comparisons do not isolate several distinctive design choices. GRPO, GSPO, and Dr. GRPO are relevant baselines, and the prefix-only ablation tests the contribution of peer-success features. However, a constant-only learned correction or a simpler leave-one-out baseline would better establish the value of the proposed reference features and coefficient fitting. Similarly, random routing with matched SFT/RL allocations would help distinguish success-based routing from simply combining SFT and RL. The connection to classical control variates is not itself a weakness; the issue is whether the added design choices improve on simpler alternatives.

  3. Downstream uncertainty is not characterized at the scale of the smaller gains. The gradient-level confidence intervals in Figure 4 are valuable, but do not establish the stability of benchmark improvements across training runs. Tables 2–4 do not report training-seed variation or benchmark confidence intervals. In particular, the full-versus-prefix-only differences on GAIA, SEAL-0, and XBench-DS correspond to net changes of three, two, and one correct answers. These results need uncertainty estimates before supporting a strong claim about the downstream value of peer-success features.

  4. The system-level comparison needs clearer separation from the 4B policy's contribution. The evaluation pipeline includes a separate 27B evidence extractor, and the paper notes that externally reported baselines may use different protocols. This does not undermine the usefulness of Apt as a system, but it limits conclusions about the relative capability of its 4B policy alone. It is particularly important to confirm that the initialization and all ablations use identical extraction, judging, and interaction budgets.

Questions for authors
  1. For each Table 4 row, what are the numbers of distinct questions, probe rollouts, SFT trajectories, and RL updates, and the total token or compute budget? A matched-budget comparison in the existing 1,024-question setting would be especially informative.

  2. How does GC-GRPO compare with a constant-only learned correction and RLOO under the same framework? For question selection, can a random-routing control preserve the SFT/RL allocation while removing dependence on probe success?

  3. Can you report repeated-seed results for the main optimizer comparison and paired uncertainty for full versus prefix-only GC-GRPO, at least on a representative training setting? Please distinguish training variability from evaluation uncertainty.

  4. Was the initial Qwen3.5-4B model evaluated with exactly the same evidence extractor, answer judge, and tool/token budgets as Apt? Which cross-system comparisons remain protocol-matched after accounting for these components?

7 comments
AuthorsOpenReview ↗

Comment 1. Training scope, total cost, and Figure 3

We thank the reviewer for recognizing the gradient construction, direct diagnostic, and system results. We clarify the training scopes and provide measured training-node times, a comparison within the Direct RL time budget, simpler-correction and random-routing controls, independent training repeats, and paired Full/Prefix uncertainty. Changes in the revised manuscript are highlighted in yellow.


The 36.27% and 48.66% full-method scores use the same method and training hyperparameters, with different amounts of training data. Section 4.3 uses 1,024 candidates for Table 2 and Figure 3. Figure 3 records training reward on the questions assigned to RL; Table 2 compares final performance on common benchmarks. Table 4's full method uses 6,400 questions. Apt follows Figure 1, generating 128 new questions per round, probing them together with retained questions, then performing SFT and fresh RL. It completed 50 rounds. The iterative ablations reuse these questions in the same order and round divisions. We will release the shared question set and code.

We acknowledge a presentation oversight in Table 4's second result row, Direct RL. It reuses Table 2's first-column results, but lacked the 1,024-question scope label. The revised Table 4 now separates it as a fixed-pool reference and specifies the training-data scopes in its caption. All other trained Table 4 variants use 6,400 questions, as do the additional K=1,2K=1,2 experiments. Section 4.5 will distinguish these scopes rather than attribute their differences solely to probing.

Thank you for highlighting the need to account for probing and SFT. We extracted elapsed training-node times covering probing, SFT, and RL to account for these costs. Both fixed-pool runs use Qwen3.5-4B initialization, GC-GRPO, Slime, and an eight-H200 node. RL uses eight questions with eight rollouts each, with four SGLang rollout GPUs and four Megatron update GPUs operating asynchronously. Adaptive uses all eight GPUs for probing and then SFT before RL.

\hline \text{Training quantity} & \text{Direct RL} & \text{Adaptive} \\\\ \hline \text{Candidate questions} & 1{,}024 & 1{,}024 \\\\ \text{SFT questions} & 0 & 353 \\\\ \text{RL questions} & 1{,}024 & 619 \\\\ \text{RL updates} & 128 & 82 \\\\ \text{Rollout output (M tokens)} & 24.8 & 42.3 \\\\ \hline \end{array}$$ Adaptive reuses successful probe trajectories for SFT, then samples fresh RL trajectories. Of the 1,024 candidates, 31 are filtered, 353 supply successful SFT demonstrations, 619 enter RL, and 21 remain for reassessment. Batching the 619 RL questions by eight within each routing batch yields 82 updates. Logged elapsed training-node times cover all three stages below. $$\begin{array}{lrr} \hline \text{Stage (elapsed hours)} & \text{Direct RL} & \text{Adaptive framework} \\\\ \hline \text{Screen/probe/reassess} & 0 & 26.9 \\\\ \text{SFT} & 0 & 8.3 \\\\ \text{RL sampling + updates} & 48.3 & 21.8 \\\\ \text{Total} & 48.3 & 57.0 \\\\ \hline \end{array}$$ Routing reduces RL updates by 35.9% and RL-stage time by 54.9% by directing some questions to demonstration learning or reassessment. Probing and SFT add time, so the completed Adaptive run takes 18.0% longer overall. To compare performance within the same training-time budget, we use Direct RL's completion at 48.3 hours as the reference and additionally evaluate the nearest available Adaptive checkpoint within that budget. This checkpoint was saved at 44.0 hours after 54 RL updates. The completed Adaptive model is also shown. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Direct (48.3 h)} & \text{Adaptive (44.0 h)} & \text{Adaptive (57.0 h)} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 78.64 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 31.28 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 33.22 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 34.23 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 75.00 & 77.00 \\\\ \hline \end{array}$$ At 44.0 hours, Adaptive is higher on three benchmarks and lower on two, including a 4.00-point gain on XBench-DS; the completed run improves four of five benchmarks.
AuthorsOpenReview ↗

Comment 2. Comparison with simpler corrections and RLOO

Full GC-GRPO outperforms the constant-only learned correction you suggested, as well as a constant-plus-peer correction, on all five benchmarks. Constant uses fi=c0f_i=c_0, and Constant + peer uses fi=c0+c1rif_i=c_0+c_1r_i, where ri=Rˉ−i−12r_i=\bar R_{-i}-\tfrac12 and Rˉ−i\bar R_{-i} is the mean reward of the other G−1G-1 trajectories for the same question. Prefix uses the five original features (1,uit,uit2,pend,it,Hit)(1,u_{it},u_{it}^{2},p_{\mathrm{end},it},H_{it}). Full adds their five products with rir_i, allowing peer success to modulate the correction; KK counts these features.

The corrected variants use the same 6,400 candidate questions and introduction order, probing and SFT procedures, and historical coefficient-fitting protocol. Thus, these controls test the specific correction features beyond Table 3's comparisons of GRPO, GSPO, and Dr. GRPO.

\hline \text{Family} & K & \text{GAIA} & \text{BrowseComp} & \text{BrowseComp-ZH} & \text{SEAL-0} & \text{XBench-DS} \\\\ & & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GRPO} & 0 & 77.67 & 45.73 & 35.29 & 35.14 & 79.00 \\\\ \text{Constant} & 1 & 79.61 & 46.68 & 36.33 & 37.84 & 79.00 \\\\ \text{Constant + peer} & 2 & 80.58 & 47.08 & 36.33 & 36.94 & 79.00 \\\\ \text{Prefix} & 5 & 82.52 & 47.95 & 46.02 & 45.95 & 81.00 \\\\ \text{Full} & 10 & \mathbf{85.44} & \mathbf{48.66} & \mathbf{47.75} & \mathbf{47.75} & \mathbf{82.00} \\\\ \hline \end{array}$$ Full exceeds Constant + peer by 11.42 points on BrowseComp-ZH and 10.81 on SEAL-0, and exceeds Prefix on all five benchmarks by 0.71 to 2.92 points. The results support conditioning the correction on generation state and its interactions with peer success beyond learning one or two coefficients. Figure 4 separately confirms positive mean gradient-MSE reductions at all 20 frozen states. **RLOO changes the reward baseline; GC-GRPO learns an additional correction while retaining GRPO's advantages, KL term, and normalization.** The comparison below makes the estimator-level distinction explicit. $$\begin{array}{lll} \hline \text{Aspect} & \text{RLOO} & \text{GC-GRPO} \\\\ \hline \text{Operation} & \text{Sequence-level reward baseline} & \text{Correction of the GRPO gradient} \\\\ \text{Core construction} & b_i=\frac{1}{G-1}\sum_{j\ne i}R_j,\quad A_i^{\mathrm{LOO}}=R_i-b_i & Y_k=X_k-\mathcal Z_k\boldsymbol c_k \\\\ \text{Conditioning} & \text{Other responses' returns for the same question} & \text{Prefix features and peer-success interactions} \\\\ \text{Estimation} & \text{Current-group mean return} & \text{Historical weighted token regression} \\\\ \hline \end{array}$$ Here, $R_i$ is RLOO's trajectory return, including its KL penalty; $X_k$ is the original GRPO gradient and $\mathcal Z_k$ stacks the zero-mean references. GC-GRPO fixes its historically fitted coefficients before current sampling. **Approximate fitting affects attainable error reduction while preserving the expected GRPO gradient**, at the sampling policy before gradient-norm clipping and the optimizer update. Constant + peer is therefore an internal GC-GRPO ablation, rather than a RLOO run. Reproducing and validating RLOO requires substantial resources and days of RL training, so we could not complete it within the discussion period and appreciate your understanding. Section 5 also discusses RLOO's leave-one-out reward baseline alongside GC-GRPO's reference construction.
AuthorsOpenReview ↗

Comment 3. Success-based routing versus random allocation

Adaptive routing outperforms random routing on all five benchmarks by 7.76 to 10.28 percentage points. The control retains the Section 4.3 initialization, 1,024-question pool, GC-GRPO, and training and evaluation settings. It randomizes which questions enter SFT or RL while keeping the numbers assigned to each stage the same. Probing still supplies successful SFT demonstrations, but the success-count intervals no longer determine assignment. Questions with no observed success may enter RL but cannot enter SFT.

\hline \text{Benchmark} & \text{Direct RL} & \text{Random routing} & \text{Adaptive routing} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 72.82 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 25.99 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 26.30 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 24.32 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 68.00 & 77.00 \\\\ \hline \end{array}$$ Section 2.2 motivates this assignment through the availability of task-reward contrast. All-success or all-failure groups have zero group-relative task-reward advantages. For eight independent binary-reward rollouts with success probability $p$, the probability of reward contrast is $1-p^8-(1-p)^8$, maximized at $p=1/2$. Random routing can allocate RL sampling to questions unlikely to provide this contrast while passing up successful demonstrations suitable for SFT. These results isolate the value of success-based assignment beyond merely combining SFT and RL. We will add Random routing to Table 2.
Reviewer_333mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adaptive training framework for long-horizon search agents that combines difficulty-aware data selection with gradient variance reduction. Questions with intermediate success rates are prioritized for RL, while successful trajectories from other difficulty regimes are used for SFT. The authors propose a control-variate-based gradient correction method that constructs zero-mean reference gradients using token-prefix features and leave-one-out rewards from other trajectories.

Critical strengths
  1. The paper's focus is that limited learning signal and high variance of policy gradients estimated from sampled trajectories, both are important to long-horizon agent training, where rollouts are computationally expensive and terminal rewards are often sparse.

  2. Rather than explicitly constructing multiple full-parameter reference gradients, the authors approximate the optimal correction coefficients through token-level regression and maintain only low-dimensional running statistics.

Critical weaknesses
  1. The adaptive training component appears to be largely built upon existing ideas in difficulty-aware curriculum learning and adaptive data selection for RL. Prior works such as EvolveSearch[1] has already explored selecting training questions according to model capability.

  2. The novelty of GC-GRPO relative to classical control-variate and baseline methods needs stronger justification. For example, [2] proposed the Optimal Token Baseline (OTB) for long-horizon LLM reinforcement learning, which also leverages token-level gradient information and computationally efficient approximations to reduce gradient variance.

  3. The contribution of the individual components is not sufficiently disentangled. The ablation results suggest that data probing accounts for most of the performance gains (e.g., BrowseComp improves from 31.51% to 48.50%), while subsequent SFT provides only marginal improvement (48.66%). A more systematic ablation independently controlling data probing, SFT, and GC-GRPO would help clarify their individual contributions and complementarity.

  4. While GC-GRPO requires only one backward pass, the complete adaptive framework introduces substantial additional computation. Each question requires eight probing trajectories, and the training procedure alternates between multiple SFT epochs and RL updates. Given the long trajectory budgets involved, these additional rollouts may represent a significant portion of the overall training cost. .

[1] EvolveSearch: An Iterative Self-Evolving Search Agent

[2] The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL

Questions for authors

What are the total rollout-token and GPU-hour costs of the adaptive framework compared with direct RL?

Does the advantage persist when the computational overhead of probing and SFT is included?

9 comments
AuthorsOpenReview ↗

Comment 1. Contribution of adaptive routing relative to EvolveSearch

We thank the reviewer for recognizing the importance of learning signals and gradient variance in long-horizon search, and the practical value of our token-regression implementation. Changes in the revised manuscript are highlighted in yellow.


Our framework makes current solvability determine the learning objective for each question before RL optimization. Section 5 discusses EvolveSearch, which alternates SFT and RL and selects accumulated RL trajectories to construct subsequent SFT data. Our additional decision is whether a current question should supply a demonstration, receive RL updates, or wait for reassessment.

\hline \text{Decision} & \text{EvolveSearch} & \text{Our framework} \\\\ \hline \text{Selected unit} & \text{Completed RL trajectories} & \text{Current questions} \\\\ \text{RL question pool} & \text{Current raw-data partition} & \text{Intermediate-solvability questions} \\\\ \text{Selection criterion} & \text{Reward and tool-call filters} & \text{Independent solvability probes} \\\\ \text{Learning use} & \text{Prepare subsequent SFT data} & \text{Assign SFT, RL, or reassessment} \\\\ \text{Core selection rule} & \mathcal D_{\mathrm{SFT}}=\mathcal F(\mathcal T_{\mathrm{RL}}) & \operatorname{route}(q)=\begin{cases} \mathrm{SFT}, & K(q)\in\\{1,2,7,8\\},\\\\ \mathrm{RL}, & 3\le K(q)\le6,\\\\ \mathrm{reassess}, & K(q)=0 \end{cases} \\\\ \hline \end{array}$$ The formula row summarizes the selection mechanisms. Here $\mathcal T_{\mathrm{RL}}$ is the accumulated RL trajectory pool, $\mathcal F$ applies EvolveSearch's reward, deduplication, and tool-call filters, and $K(q)$ counts successful probes out of eight attempts. **By combining probing with SFT, we provide direct supervision where GRPO's within-group reward contrast can be weak or absent.** GRPO computes task-reward advantages by comparing outcomes within a question group; all-success or all-failure groups therefore yield zero task-reward advantages. Our framework addresses this constraint by using successful probe trajectories from low- and high-solvability questions for SFT. These demonstrations reinforce successful behavior directly, without requiring a mixed-reward group. Intermediate-solvability questions receive fresh RL rollouts after SFT, while questions with no observed success are retained for reassessment (Section 2.2). The controls in Comment 3 test this assignment of questions to learning objectives.
AuthorsOpenReview ↗

Comment 2. Contribution of GC-GRPO relative to classical control variates and OTB

GC-GRPO adds a learned noise correction to the implemented GRPO gradient while keeping its expectation unchanged even when fitting is approximate. The contribution is the GRPO-specific reference construction and fitting protocol. We share the control-variate foundation of OTB; the concrete distinction is what is corrected and how the correction is estimated.

\hline \text{Aspect} & \text{OTB} & \text{GC-GRPO} \\\\ \hline \text{Fitting target} & \text{Reward-to-go} & \text{Original GRPO token weights} \\\\ \text{Construction} & \text{Cumulative-energy token baseline} & \text{Zero-mean prefix/peer references} \\\\ \text{Estimation} & \text{Current-group weighted returns} & \text{Historical token regression} \\\\ \text{Core construction} & \displaystyle\widehat B_t=\frac{\sum_iG_t^{(i)}\widehat W_t^{(i)}}{\sum_i\widehat W_t^{(i)}} & \begin{gathered} Y_k=X_k-\mathcal Z_k\boldsymbol c_k,\\\\ \mathbb E[Y_k]=\mathbb E[X_k] \end{gathered} \\\\ \hline \end{array}$$ Here $G_t^{(i)}$ is reward-to-go and $\widehat W_t^{(i)}$ is cumulative logit-gradient energy. For GC-GRPO, $X_k$, $Y_k$, and $\mathcal Z_k$ denote the original GRPO gradient, corrected gradient, and stacked zero-mean references. GC-GRPO retains GRPO's group-relative advantages, KL term, and normalization. Predictable prefix features and leave-one-out peer rewards construct zero-mean references, while output-layer-weighted token regression fits their coefficients $\boldsymbol c_k$ on preceding batches. Fixing these coefficients before current sampling separates mean preservation from fitting accuracy. **Approximate fitting affects the attainable error reduction while the expected GRPO gradient remains unchanged**, at the sampling policy before gradient-norm clipping and the optimizer update. This allows a tractable fit without requiring the exact full-gradient optimum. The correction needs one backward pass and only low-dimensional running statistics (Sections 3.1 and 3.2). **The ablations test the state dependence and peer interactions used by this correction.** Constant uses $f_i=c_0$; Constant + peer uses $f_i=c_0+c_1r_i$, where $r_i=\bar R_{-i}-\tfrac12$ and $\bar R_{-i}$ is the mean reward of the other $G-1$ trajectories for the same question. Prefix uses five prefix features; Full adds their five peer-weighted counterparts. All corrected variants use historical fitting, with probing and SFT fixed. BC, BC-ZH, and XB denote BrowseComp, BrowseComp-ZH, and XBench-DS.
AuthorsOpenReview ↗

Comment 3. Contributions of probing, SFT, and gradient correction

We separate the benefit of success-based question assignment from the benefit of SFT itself. To test the assignment rule, we added a random-routing control under the Section 4.3 setting, with the same 1,024-question pool, initialization, GC-GRPO, and training and evaluation settings. It randomizes question identities while retaining the same numbers of SFT and RL questions. Probing still supplies successful SFT demonstrations, but the success-count intervals no longer determine the assignment. Questions with no observed success may enter RL but cannot enter SFT.

\hline \text{Benchmark} & \text{Direct RL} & \text{Random routing} & \text{Adaptive routing} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 72.82 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 25.99 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 26.30 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 24.32 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 68.00 & 77.00 \\\\ \hline \end{array}$$ **Adaptive routing improves all five benchmarks over Random by 7.76 to 10.28 points.** This isolates the value of using current solvability to assign learning objectives beyond merely including SFT and RL. **With the routing procedure and GC-GRPO held fixed, SFT provides additional gains on every benchmark.** The following existing Table 4 comparison uses the iterative configurations with the same 6,400 candidate questions and introduction order, retaining probing and Full GC-GRPO while varying SFT. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Without SFT} & \text{With SFT} & \text{Gain} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{pp} \\\\ \hline \text{GAIA} & 80.58 & 85.44 & 4.86 \\\\ \text{BrowseComp} & 48.50 & 48.66 & 0.16 \\\\ \text{BrowseComp-ZH} & 44.64 & 47.75 & 3.11 \\\\ \text{SEAL-0} & 45.95 & 47.75 & 1.80 \\\\ \text{XBench-DS} & 80.00 & 82.00 & 2.00 \\\\ \hline \end{array}$$ The 0.16-point improvement on BrowseComp is therefore not representative of SFT's contribution across benchmarks. It adds 4.86 points on GAIA and 3.11 on BrowseComp-ZH, with positive gains on the other two benchmarks as well. Finally, Table 3 holds the probing and SFT procedures common while changing the RL algorithm. Replacing GRPO with GC-GRPO improves all five benchmarks by 2.93 to 12.61 points, as also shown by the GRPO and Full rows in Comment 2. These comparisons assess assignment, SFT, and gradient correction within their respective matched training settings. We will add Random routing to Table 2 and make these component comparisons explicit in Section 4.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.