acceptodds
Withdrawn from ICLR 2027

ASCT: Agentic Search Critical Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) holds great promise for training autonomous agents in sequential decision-making, yet outcome-based methods face a fundamental credit assignment problem in long-horizon interactions: sparse rewards provide no signal about which intermediate actions were effective. Existing remedies are costly. Process Reward Models (PRMs) require training and running an auxiliary model to score every intermediate step at inference time, incurring substantial runtime overhead. Reward shaping relies on hand-engineered multi-objective functions, and value-based credit assignment suffers from unstable value estimation over long horizons. We propose Agentic Search Critical Training (ASCT), a two-stage RL framework built on a simple insight: teaching an agent to critically judge which candidate action is better and to reason about why it is better yields a higher performance ceiling than teaching it to imitate correct answers. Rather than scoring steps with an auxiliary model at runtime, ASCT decomposes multi-step search trajectories into three hierarchical perception levels—environment cognition, strategic decision-making, and tactical execution—and constructs multiplechoice discriminative tasks at each level via trajectory truncation. In Stage 1, hierarchical rewards inject these critical priors in a controlled setting; in Stage 2, all scaffolding is removed and the agent explores autonomously with only outcome rewards, leveraging its internalized analytical capabilities. Experiments on multi-hop QA and DeepResearch benchmarks show that ASCT consistently outperforms GRPO and other strong baselines across multiple model scales.

Primary area:
reinforcement learning
Keywords:
LLM, Agent, RL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_o9EAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets sparse outcome rewards in multi-turn search agent RL. It proposes ASCT, a two-stage GRPO pipeline. In Stage 1, expert trajectories taken from public Tool-Star and WebSailor resources are truncated at intermediate steps, and GLM-5.2 annotates each step at three levels: observation type (L1), strategy (L2), and a 4-way choice between the expert’s actual query/URL and three generated distractors (L3). The policy is trained with GRPO on these multiple-choice tasks using a weighted 0.1/0.2/0.7 reward. In Stage 2, all scaffolding is removed and the same policy is trained with outcome-only GRPO. Experiments use Qwen2.5-7B on Search-R1’s multi-hop QA suite and Qwen3-8B/14B on GAIA, WebWalkerQA, HLE, and XBench, reporting small but consistent gains over Search-R1 and AEPO.

Critical strengths

The idea of warming up a search agent with verifiable discriminative tasks before outcome RL is simple, and the pipeline is described clearly. The appendix gives full prompts and hyperparameters, and the Stage 1 vs. Stage 2 training curves are informative.

Critical weaknesses

The novelty is limited. Converting expert trajectories into step-level supervision, or identifying critical steps before RL, is by now fairly standard practice (ATLaS, ConSpec, CARL), and the two-stage “discriminate the better action, then RL” recipe is essentially ACT. The main addition is the three-level decomposition, and the ablation suggests that L3 does almost all of the work. L3 is just picking the expert’s own action out of four options, which is closer to discriminative behavior cloning than to “critical judgment”. Despite motivating the work by sparse rewards, the paper never compares against any process-reward or step-level credit method for search agents (StepSearch, MT-GRPO, IGPO, Tree-GRPO/BranPO, SAPO, etc.).

The baselines are weak and dated. Table 1 is essentially Search-R1-era methods, with numbers copied from that paper even though ASCT uses Google Search top-5 rather than the local Wikipedia retriever. Table 2 copies all numbers from AEPO while using a different judge (GLM-5.2), so neither comparison is controlled. The backbone choice is also odd: Qwen2.5-7B for QA but Qwen3-8B/14B for DeepResearch, with no justification and no cross-check.

The gains are 1–2 points on 100-question subsets, i.e., one or two questions, with single-run Pass@1 and no variance reported. Stage 1 also adds about 240 GRPO steps of extra compute that the GRPO baseline does not get. Finally, the “w/o Stage 1” ablation (GAIA 32.0) is described as the authors’ own GRPO run, yet it is identical to the GRPO row copied from AEPO. This needs clarification, because the headline 14.6-point gain from Stage 1 depends on it.

Questions for authors
  • How does ASCT compare, under the same retriever, judge, data, and compute, with step-level or process-reward methods for search agents (e.g., StepSearch, MT-GRPO, IGPO, Tree-GRPO), and with ACT applied directly to this setting?

  • Is the “w/o Stage 1” GAIA number (32.0) from your own GRPO run, or taken from AEPO? If it is your own, why do all per-split numbers match AEPO exactly? Please also report a compute-matched GRPO baseline (same total steps) with multiple seeds.

  • Why use Qwen2.5-7B for multi-hop QA but Qwen3 for DeepResearch? Do the conclusions hold if the same backbone is used across both task families?

Reviewer_3hhKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Agentic Search Critical Training (ASCT) for long-horizon search agent training. The motivation is that decision making in search agent can be decomposed into hierarchical cognitive levels. The proposed method first trains the agent to discriminate the quality of strategic decisions, and then utilizes this ability to improve autonomous exploration. ASCT segments expert trajectories and construct three different levels of cognitive choice tasks, which are then used for GRPO training with a hierarchical proportional reward scheme. After that the agent is trained normally and autonomously with GRPO and outcome based reward on search tasks. The paper reports experiment results on several multi-hop QA and search agentic tasks, and ASCT performs better than GRPO under evaluated tasks and model sizes.

Critical strengths

None.

Critical weaknesses
  1. Presentation. The paper is poorly written and under-polished. The writing is vague, repetitive and lack a clear logical structure. It is difficult for me to follow the paper or to understand the proposed method despite careful reading. I strongly encourage authors to check and polish AI generated content before submitting the paper.

  2. Motivation and Soundness. The proposed method is not well motivated. It is unclear why such a two-stage training pipeline is needed. The L1/L2/L3 decomposition in the first stage along with the hierarchical reward design are unjustified and rely on heuristic design. The motivations and challenges (credit assignment, sparse reward) presented in the Introduction appear mismatched with the proposed method, since ASCT still relies on standard credit assignment based on outcome reward throughout.

Questions for authors

See weaknesses

Reviewer_kvcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASCT (Agentic Search Critical Training), which is a two-stage reinforcement-learning framework that teaches search agents to evaluate intermediate decisions before training them to solve complete tasks using sparse final-answer rewards. It converts truncated expert search trajectories into discriminative tasks at three levels: environment cognition—judging whether retrieved information is useful, redundant, conflicting, or irrelevant; strategic decision-making—deciding whether to continue searching, visit a page, stop, or backtrack; and tactical execution—selecting a concrete query or URL against realistic distractors. Stage 1 uses GRPO with weighted rewards for correct judgments, emphasizing tactical action selection. Stage 2 removes the candidate options and explicit hierarchical judgments, then trains the same policy through autonomous search using only outcome rewards. The central idea is to internalize intermediate decision skills into the policy’s initialization, supporting subsequent exploration without a separate process reward model. Empirical results on QA and Deepresearch benchmarks show promising results.

Critical strengths
  1. The paper’s critical strength is its practical approach to preparing agents for autonomous search through structured supervision of intermediate decisions. ASCT converts existing expert trajectories into concrete tasks covering information assessment, search strategy, and query or URL selection, making the good search behavior learning directly trainable.
Critical weaknesses
  1. From the experiments in the paper, I am not convinced that the paper isolates “critical judgment” as the source of the performance gains. ASCT adds approximately 240 Stage-1 GRPO steps on 10K expert-derived examples before the 300-step autonomous training stage, introducing both additional supervision and training relative to outcome-only GRPO. Although the paper includes SFT+GRPO learning curves, it does not establish a clearly matched comparison. The ablations study supports the usefulness of the complete curriculum, but stronger causal evidence would require comparisons against same-data SFT and discrimination without explanations, under matched training budgets. I hope the authors clarify that in the rebuttal.
Questions for authors

In Table 2, the paper imports baseline numbers from AEPO, the actual runs use live Google search and GLM-5.2 judging. Matching questions does not establish matching retrieved content, page availability, judge behavior, or interaction conditions. Can you please clarify why this is the case? Also do you think re-running the strongest baselines under the same evaluation environment would materially strengthen the comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.