- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses two challenges in training search agents with reinforcement learning: selecting informative training questions and reducing noise in policy gradients. The authors propose an adaptive training framework that generates evidence-grounded questions and routes them to SFT, RL, or later reassessment based on the agent's current success rate. They further introduce GC-GRPO, which uses zero-mean reference gradients constructed from token-prefix features and peer-trajectory rewards to reduce gradient estimation error while preserving the expected GRPO gradient at the sampling policy.
Critical strengths
- GC-GRPO provides an interesting application of control variates to group-relative policy optimization. I particularly appreciate that the authors go beyond benchmark performance and directly measure gradient estimation error in Figure 4. This provides useful evidence for the proposed mechanism.
- Apt consistently improves over its initial model across five benchmarks, with competitive performance against existing search agents. The ablations in Tables 3 and 4 also provide evidence that both adaptive question selection and gradient correction contribute to the improvements.
Critical weaknesses
- The training-cost comparison is not fully controlled. In Section 4.3, the adaptive framework is compared against direct RL using fewer RL steps. However, the adaptive framework additionally requires multiple search rollouts for question probing and SFT training. These costs are not accounted for in the comparison. Consequently, it is unclear whether the adaptive framework is more efficient under the same overall compute or trajectory budget. A compute-matched comparison would make the claimed efficiency benefits more convincing.
- Limited evidence for the robustness and generalizability of the improvements. The experiments are conducted with a single 4B backbone, and no results across multiple training seeds are reported. Some improvements over competing RL algorithms are relatively small, particularly given that several evaluation benchmarks contain only around 100 questions. Moreover, the search system uses a separate 27B model for evidence extraction, making comparisons with externally reported search-agent results difficult to interpret without matched tool configurations.
Questions for authors
- Could the authors provide a comparison under matched overall training costs, including question probing, SFT, and RL? It would also be useful to compare against random or fixed question routing with the same SFT/RL allocation to better isolate the benefit of adaptive selection.
- How stable are the reported benchmark improvements across different training seeds? Could the authors also clarify the contribution of the 27B evidence extraction service to the reported performance, particularly when comparing Apt with other 4B search agents?
7 comments
Comment 2. Adaptive selection versus random question routing
Success-based routing outperforms random routing on all five benchmarks by 7.76 to 10.28 percentage points. The control retains the Section 4.3 candidate pool, initialization, GC-GRPO, and training and evaluation settings. It randomizes which questions enter SFT or RL while keeping the numbers of questions assigned to each stage the same. The same probing procedure supplies successful demonstrations for SFT, but its success-count intervals no longer determine the assignment. SFT requires a successful demonstration; questions with no observed success may enter RL but cannot enter SFT.
\hline \text{Benchmark} & \text{Direct RL} & \text{Random routing} & \text{Adaptive routing} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 72.82 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 25.99 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 26.30 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 24.32 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 68.00 & 77.00 \\\\ \hline \end{array}$$ Section 2.2 explains why objective assignment matters. All-success or all-failure rollout groups have zero group-relative task-reward advantages. For eight independent binary-reward rollouts with success probability $p$, the probability of reward contrast is $1-p^8-(1-p)^8$, maximized at $p=1/2$. Random allocation can therefore spend RL sampling on questions less likely to provide reward contrast, while passing up successful demonstrations that could provide direct SFT supervision. This issue also motivates DAPO's dynamic sampling and query recycling for agentic search. The results support matching questions to learning objectives beyond simply combining SFT and RL. Random routing is now included in Table 2, and Section 2.2 of the revised manuscript cites DAPO alongside query recycling.Comment 3. Stability across training seeds
Following your suggestion, we used our available training resources to complete two additional independent seed runs under the Section 4.3 settings. This experiment uses our complete adaptive framework with GC-GRPO on a smaller pool of 1,024 training questions. Apt in the main comparison is trained with 6,400 questions; the routing, SFT, and GC-GRPO components are retained in these repetitions.
Seed 1 denotes the Table 2 Adaptive run; Seeds 2 and 3 are the additional independent training runs, with the initialization, candidate pool, training settings, and tools held the same. Mean and sample SD are computed across the three seeds.
\hline \text{Benchmark} & \text{Seed 1} & \text{Seed 2} & \text{Seed 3} & \text{Mean} & \text{SD} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{pp} \\\\ \hline \text{GAIA} & 80.58 & 79.61 & 81.55 & 80.58 & 0.97 \\\\ \text{BrowseComp} & 36.27 & 36.97 & 35.31 & 36.18 & 0.83 \\\\ \text{BrowseComp-ZH} & 35.99 & 37.02 & 34.95 & 35.99 & 1.04 \\\\ \text{SEAL-0} & 33.33 & 33.33 & 31.53 & 32.73 & 1.04 \\\\ \text{XBench-DS} & 77.00 & 81.00 & 79.00 & 79.00 & 2.00 \\\\ \hline \end{array}$$ All three runs exceed Direct RL on GAIA, BrowseComp, BrowseComp-ZH, and XBench-DS, and remain lower on SEAL-0. **The direction of the framework's benchmark gains is reproduced across seeds, with SDs of 0.83 to 1.04 points on four benchmarks and 2.00 points on XBench-DS.** The Section 4.3 Adaptive run takes 57.0 hours on the eight-H200 node, while full Apt training takes less than a week. These repetitions assess the same proposed method at a smaller training-data scale. Given our limited training resources, we prioritized the two additional seed runs and appreciate your understanding that we could not train additional backbones within the discussion period.
Comment 1. Training time including probing, SFT, and RL
We thank the reviewer for recognizing adaptive question selection, GC-GRPO, and the direct gradient diagnostic. Below, we provide complete training-node timing, a comparison within a common time budget, a random-routing control, independent training repeats, and direct tests of GC-GRPO and the page-summary service. Changes in the revised manuscript are highlighted in yellow.
The two settings in Section 4.3 share the initialization, candidate questions, hardware, and RL implementation, and differ in how questions are used for training. Their configurations are summarized below.
\hline \text{Configuration} & \text{Direct RL} & \text{Adaptive framework} \\\\ \hline \text{Initialization} & \text{Qwen3.5-4B} & \text{Qwen3.5-4B} \\\\ \text{Candidate questions} & 1{,}024 & 1{,}024 \\\\ \text{Training node} & \text{8 H200 GPUs} & \text{8 H200 GPUs} \\\\ \text{Training framework} & \text{Slime} & \text{Slime} \\\\ \text{RL algorithm} & \text{GC-GRPO} & \text{GC-GRPO} \\\\ \text{RL batch} & \text{8 questions}\times\text{8 rollouts} & \text{8 questions}\times\text{8 rollouts} \\\\ \text{RL rollout/update GPUs} & 4/4 & 4/4 \\\\ \text{Screening/probing} & \text{No} & \text{Yes} \\\\ \text{SFT questions} & 0 & 353 \\\\ \text{RL questions} & 1{,}024 & 619 \\\\ \text{RL updates} & 128 & 82 \\\\ \hline \end{array}$$ Both RL stages use four GPUs for SGLang rollout and four for Megatron updates, with asynchronous overlap. Adaptive training uses all eight GPUs during probing and SFT. Of the 1,024 candidates, 31 are filtered, 353 supply successful SFT demonstrations, 619 enter RL, and 21 remain for reassessment. The 619 RL questions yield 82 updates when batched by eight within each routing batch. We extracted elapsed training-node times from the training logs, covering screening/probing, SFT, and RL sampling and updates. $$\begin{array}{lrr} \hline \text{Stage (elapsed hours)} & \text{Direct RL} & \text{Adaptive framework} \\\\ \hline \text{Screen/probe/reassess} & 0 & 26.9 \\\\ \text{SFT} & 0 & 8.3 \\\\ \text{RL sampling + updates} & 48.3 & 21.8 \\\\ \text{Total} & 48.3 & 57.0 \\\\ \hline \end{array}$$ Adaptive training reduces RL updates by 35.9% and RL-stage time by 54.9%; including probing and SFT, its completed run takes 18.0% longer overall. To compare performance at comparable training times, we use Direct RL's completion time of 48.3 hours as the reference and evaluate the nearest available Adaptive checkpoint within that budget. This checkpoint was saved at 44.0 hours, after 54 RL updates. The completed Adaptive model at 57.0 hours is also shown. $$\begin{array}{lrrr} \hline \text{Benchmark} & \text{Direct (48.3 h)} & \text{Adaptive (44.0 h)} & \text{Adaptive (57.0 h)} \\\\ & \text{ACC (\\%)} & \text{ACC (\\%)} & \text{ACC (\\%)} \\\\ \hline \text{GAIA} & 77.67 & 78.64 & 80.58 \\\\ \text{BrowseComp} & 31.51 & 31.28 & 36.27 \\\\ \text{BrowseComp-ZH} & 32.87 & 33.22 & 35.99 \\\\ \text{SEAL-0} & 36.03 & 34.23 & 33.33 \\\\ \text{XBench-DS} & 71.00 & 75.00 & 77.00 \\\\ \hline \end{array}$$ The 44.0-hour checkpoint is higher on three benchmarks and lower on two than Direct RL, with a 4.00-point gain on XBench-DS. Completing the adaptive schedule improves four of five benchmarks over Direct RL.