acceptodds
Under review as a conference paper at ICLR 2027

Process-Supervised Learning for Agentic Search via Dual-Granularity Advantage

Abstract

Reinforcement learning (RL) has emerged as a promising paradigm for optimizing search agents in complex reasoning tasks. However, traditional outcome-based RL often relies on sparse terminal rewards and suffers from credit assignment challenges, resulting in gradient starvation and high-variance policy updates. This limits optimization stability and can lead to process hallucinations, where agents achieve correct answers via flawed reasoning or redundant retrieval. While recent process-aware approaches attempt to mitigate these issues, they often lack on-policy exploration capability or collapse step-level credit into coarse trajectory-level optimization. To address these limitations, we propose ProSearch, a dual-granularity process-supervised RL framework. Leveraging a Monte Carlo Tree Search (MCTS)-derived Process Reward Model (PRM), ProSearch introduces a novel dual-granularity advantage mechanism that combines dense step-level signals with sparse outcome rewards. This provides precise credit assignment for specific actions, mitigating gradient starvation while theoretically improving the Signal-to-Noise Ratio (SNR). Extensive experiments on five multi-hop QA benchmarks demonstrate that ProSearch achieves overall superior performance compared to strong baselines, particularly on complex long-horizon tasks. Further fine-grained analysis reveals that ProSearch not only stabilizes training dynamics but also significantly reduces logical fallacies and redundant exploration.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agentic Search, Process Reward Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WbcXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an RL algorithm for training search agents that uses fine-grained credit assignment, process rewards, and process-level advantage assignment to improve their performance.

Critical strengths

Fine-grained credit assignment for retrieval actions is a timely topic. The proposed method is reasonable and feasible.

Critical weaknesses
  1. Qwen3-8B is not a strong baseline for search agents, so the choice of backbone is not representative. The authors should evaluate the gains using backbones such as Qwen3.5 or Qwen3.6 that have undergone targeted agent training.

  2. The naïve Qwen3-8B baseline achieves particularly low scores on some benchmarks, such as HotpotQA, which is inconsistent with my expectations based on previous results. I would like the authors to clarify the experimental setup.

  3. Process rewards and process-level credit assignment are not particularly novel in RL research and have already been explored in previous work on training search agents. The novelty of this paper is therefore limited.

  4. The ablation comparisons, such as those against outcome-based methods, do not control for total computational cost, including PRM training.

Questions for authors

See weaknesses.

Reviewer_jk99OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ProSearch, a process-supervised RL framework for agentic search that combines step-level rewards from a PRM with trajectory-level outcome rewards through a dual-granularity advantage formulation. The PRM is trained using MCTS-generated sibling preference pairs, followed by PRM-guided policy refinement and online GRPO optimization. Experiments on five QA benchmarks demonstrate improvements over several baselines, with additional analyses of reasoning quality and training stability.

Critical strengths
  1. The paper addresses the credit assignment problem in multi-turn agentic search, where sparse outcome rewards cannot directly distinguish useful and unhelpful intermediate actions.
  2. ProSearch achieves the highest average F1 across five QA benchmarks, with notable improvements on 2Wiki and MuSiQue compared with Search-R1.
Critical weaknesses
  1. The proposed advantage is a weighted combination of outcome and process advantages, which is similar to existing process-supervised RL methods. Moreover, since process rewards are one of the main contributions, the authors should compare the trained PRM against existing PRMs or simpler step-scoring heuristics, such as entropy or answer log-probabilities, to demonstrate its effectiveness.
  2. MCTS is computationally expensive, requiring 200 simulations per query according to the paper. This makes PRM training potentially costly and difficult to scale to long-horizon tasks such as SWE-bench or Terminal-Bench, where trajectories can involve around 100 turns. The authors should discuss the computational overhead and whether the proposed approach can scale to these more complex settings.
Questions for authors
  1. The PRM is trained using pairwise preferences, but its scores are subsequently used for absolute thresholding and cross-state normalization without evidence of calibration. How do the authors ensure that these scores are comparable across different states? Moreover, intermediate steps that appear unsuccessful may still contribute to eventual success through exploration or error recovery. How does the proposed reward formulation account for such actions and avoid incorrectly penalizing them?
Reviewer_WH9EOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ProSearch, a process-supervised reinforcement learning framework for agentic search. It uses MCTS to generate candidate actions from the same state and trains a process reward model (PRM) using LLM-annotated preferences. Following PRM-guided policy refinement, it combines step-level process advantages with trajectory-level outcome advantages in GRPO. Experiments with Qwen3-8B on five QA benchmarks evaluate answer accuracy, reasoning quality, and retrieval efficiency.

Critical strengths
  1. Clear and practical process-supervision design. Comparing candidate actions under the same context provides targeted supervision for local decisions, while separate process advantages allow different steps to receive distinct feedback. The frozen PRM also avoids repeated calls to an external judge during online training. The framework is straightforward and explicitly addresses credit assignment.

  2. Empirical gains in both accuracy and retrieval efficiency. ProSearch achieves an average F1 of 49.2, compared with 46.7 for Search-R1. On a matched evaluation subset, it reduces average retrieval calls from 2.53 to 1.64 while improving F1. Further component ablations, a comparison with aggregated process rewards, and on-policy PRM evaluation provide reasonably broad empirical support.

Critical weaknesses
  1. Local rewards may favor plausible queries without directly rewarding effective search. The PRM Rϕ(st,at)R_{\phi}(s_t,a_t) does not observe the documents returned by the current search, and subsequent process rewards are not directly credited to earlier search actions. Consequently, the additional process signal may favor logically reasonable queries without distinguishing whether they acquire necessary evidence or enable useful subsequent steps. Outcome rewards provide indirect feedback, but become uninformative for distinguishing trajectories when all outcomes within a group receive the same reward. Evaluating failed-retrieval recovery, query reformulation, and searches with delayed benefits would help determine whether this design improves actual search decisions.

  2. Process improvements coincide with a higher proportion of trajectories classified as factual hallucinations. In Figure 4, this proportion increases relative to Search-R1 on all three datasets: 10.1% → 30.8% on HotpotQA, 17.4% → 20.0% on PopQA, and 13.1% → 17.4% on MuSiQue. Although correct trajectories increase and logical fallacies decrease, this unfavorable change in the error profile deserves explicit analysis.

Questions for authors
  1. How was the threshold (R_phi(s,a)>0) determined?

  2. Do actions preferred by the PRM actually lead to higher downstream returns?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.