- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper targets sparse outcome rewards in multi-turn search agent RL. It proposes ASCT, a two-stage GRPO pipeline. In Stage 1, expert trajectories taken from public Tool-Star and WebSailor resources are truncated at intermediate steps, and GLM-5.2 annotates each step at three levels: observation type (L1), strategy (L2), and a 4-way choice between the expert’s actual query/URL and three generated distractors (L3). The policy is trained with GRPO on these multiple-choice tasks using a weighted 0.1/0.2/0.7 reward. In Stage 2, all scaffolding is removed and the same policy is trained with outcome-only GRPO. Experiments use Qwen2.5-7B on Search-R1’s multi-hop QA suite and Qwen3-8B/14B on GAIA, WebWalkerQA, HLE, and XBench, reporting small but consistent gains over Search-R1 and AEPO.
Critical strengths
The idea of warming up a search agent with verifiable discriminative tasks before outcome RL is simple, and the pipeline is described clearly. The appendix gives full prompts and hyperparameters, and the Stage 1 vs. Stage 2 training curves are informative.
Critical weaknesses
The novelty is limited. Converting expert trajectories into step-level supervision, or identifying critical steps before RL, is by now fairly standard practice (ATLaS, ConSpec, CARL), and the two-stage “discriminate the better action, then RL” recipe is essentially ACT. The main addition is the three-level decomposition, and the ablation suggests that L3 does almost all of the work. L3 is just picking the expert’s own action out of four options, which is closer to discriminative behavior cloning than to “critical judgment”. Despite motivating the work by sparse rewards, the paper never compares against any process-reward or step-level credit method for search agents (StepSearch, MT-GRPO, IGPO, Tree-GRPO/BranPO, SAPO, etc.).
The baselines are weak and dated. Table 1 is essentially Search-R1-era methods, with numbers copied from that paper even though ASCT uses Google Search top-5 rather than the local Wikipedia retriever. Table 2 copies all numbers from AEPO while using a different judge (GLM-5.2), so neither comparison is controlled. The backbone choice is also odd: Qwen2.5-7B for QA but Qwen3-8B/14B for DeepResearch, with no justification and no cross-check.
The gains are 1–2 points on 100-question subsets, i.e., one or two questions, with single-run Pass@1 and no variance reported. Stage 1 also adds about 240 GRPO steps of extra compute that the GRPO baseline does not get. Finally, the “w/o Stage 1” ablation (GAIA 32.0) is described as the authors’ own GRPO run, yet it is identical to the GRPO row copied from AEPO. This needs clarification, because the headline 14.6-point gain from Stage 1 depends on it.
Questions for authors
-
How does ASCT compare, under the same retriever, judge, data, and compute, with step-level or process-reward methods for search agents (e.g., StepSearch, MT-GRPO, IGPO, Tree-GRPO), and with ACT applied directly to this setting?
-
Is the “w/o Stage 1” GAIA number (32.0) from your own GRPO run, or taken from AEPO? If it is your own, why do all per-split numbers match AEPO exactly? Please also report a compute-matched GRPO baseline (same total steps) with multiple seeds.
-
Why use Qwen2.5-7B for multi-hop QA but Qwen3 for DeepResearch? Do the conclusions hold if the same backbone is used across both task families?