acceptodds
Under review as a conference paper at ICLR 2027

Beyond Outcome Rewards: A Study of Intermediate Supervision for Search Agents

Abstract

Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.

Primary area:
reinforcement learning
Keywords:
Search Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xT5yOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how intermediate retrieval rewards affect RL training for search agents. The authors construct multi-hop QA data with annotated supporting evidence and compare three reward signals: evidence coverage (Cov), dependency-aware coverage (Cov-Dep), and answer matching (AM). They also investigate two ways of incorporating these signals into training: adding them to the trajectory-level reward or assigning credit directly to individual search actions. Experiments on seven QA benchmarks show that intermediate supervision improves performance, with local credit assignment generally working better than scalar rewards. The paper also examines action-reward alignment and how intermediate supervision affects evidence acquisition.

Critical strengths
  1. The paper provides a systematic comparison of different intermediate reward signals and credit assignment strategies. The controlled experimental setup makes it easier to understand how these design choices affect search-agent training.

  2. I find the analysis of action-level credit assignment interesting. In particular, the reward permutation experiment provides fairly convincing evidence that assigning retrieval credit to the corresponding search action matters, especially for multi-hop questions.

Critical weaknesses
  1. Some of the insights are supported by rather small performance differences. For example, Cov-Dep-scalar achieves 52.82 average F1, compared with 52.64 for Cov-scalar. Without statistical significance tests or repeated runs, I am not convinced that these differences are meaningful enough to support conclusions about the reward design.

  2. The comparison between scalar rewards and local credit assignment could be better controlled. The two approaches also differ in how the additional rewards are normalized and incorporated into the policy update. It would be helpful to compare them under similar update magnitudes to better isolate the effect of credit assignment.

  3. All experiments use a single model backbone and a fixed retrieval setup. The training data also relies on constructed questions with annotated evidence. I would like to see whether the findings hold with other model sizes or more realistic search environments.

Questions for authors

See Weaknesses

Reviewer_KN6JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how intermediate retrieval rewards interact with credit assignment when training search agents. It constructs 14,000 corpus-grounded questions with evidence dependency graphs and compares answer matching, evidence coverage, and dependency-constrained coverage. Each signal is incorporated either into a scalar trajectory reward or as a signed residual advantage on executed query tokens. Experiments train Qwen3-4B-Instruct-2507 with GRPO using a fixed retriever over the KILT corpus. On seven QA benchmarks, local coverage credit improves average F1 from 51.25 to 54.34. Additional interventions permute credit across actions and restrict it to different outcome groups, while an annotated development set supports analysis of evidence acquisition.

Critical strengths
  1. A useful experimental decomposition of intermediate supervision. Comparing three signals under two incorporation rules, while keeping the model, training data, retriever, and training budget fixed, is more informative than testing a single reward against outcome-only training. Cov and Cov-Dep share the same grounding and differ specifically in prerequisite closure, which makes their comparison particularly interpretable. Cov-local improves F1 over outcome-only training on all seven benchmarks, and the aggregate benefit survives removal of any one benchmark. This provides evidence that the improvement is not driven by a single dataset.

  2. The action-permutation intervention directly tests the value of assigning credit to its source action. The control preserves the eligible actions and the multiset of signed event masses while changing which query receives each mass. Aligned credit improves average F1 over this control by 2.12 points for coverage and 3.12 points for answer matching, with larger differences on multi-hop QA. This is a substantive diagnostic beyond a standard component-removal ablation and supports the importance of event-action correspondence within the proposed local formulation.

Critical weaknesses
  1. The evaluation does not yet establish the robustness of the central empirical conclusions. Section 4 explicitly states that the seven-benchmark suite informed development decisions, but does not explain which decisions used it or provide an independent benchmark evaluation after the configuration was fixed. The separate synthetic development split is used for behavioral analysis rather than independent validation of the benchmark gains. Moreover, no repeated-training results or uncertainty intervals are reported. This is especially consequential for the claimed benefit of dependency constraints under scalar incorporation, which is only 0.18 F1, and the advantage of using all outcome groups over no-flat coverage, which is only 0.32 F1. The leave-one-dataset-out analysis checks mixture sensitivity, but cannot resolve adaptive selection or training variability. The larger improvements are promising; the finer design recommendations need stronger evidence.

  2. Several comparisons change the amount and normalization of supervision along with where it is assigned. Scalar incorporation uses a fixed coefficient of 0.25 before trajectory-level standardization, whereas local incorporation uses separately standardized search events with a coefficient of 0.5 and payload-length-dependent event mass. These settings do not match effective update strength. Likewise, flat-only and no-flat remove local credit without rescaling what remains. Consequently, these experiments compare complete update rules, but do not isolate how much of the gain comes from localization or outcome-group coverage rather than the strength of the auxiliary update. Appendix A.4 checks scalar reward rankings on saved trajectories; it does not establish training-time robustness to reward scale. Section 7 acknowledges this limitation, but it bears directly on the paper's central interpretation. The permutation control addresses action correspondence within the local rule, not this scalar-versus-local confound.

Questions for authors
  1. Clarification and independent validation. Which development decisions used the seven-benchmark evaluation suite? Can the final fixed configuration and the main baselines be evaluated on an untouched set of questions, with the development and evaluation boundaries stated explicitly?

  2. Targeted uncertainty analysis. Do repeated training seeds support the dependency-constraint ranking reversal and the all-groups versus no-flat conclusion? Please distinguish variability across training runs from paired evaluation uncertainty, particularly for the 0.18 and 0.32 F1 differences.

  3. Targeted control of update strength. Does local incorporation remain preferable under a modest coefficient sweep or comparable auxiliary-update magnitudes? For the outcome-group restrictions, does rescaling the retained credit change the ranking? These controls would help determine which interpretation of the gains is supported.

Reviewer_puemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper constructs 14,000 QA examples annotated with subquestions, supporting evidence, and dependency structures. Within GRPO, it compares three retrieval signals—evidence coverage (Cov), dependency-aware coverage (Cov-Dep), and answer matching—and two ways of incorporating them: trajectory-level scalar rewards and local advantages assigned to executed query tokens. The best-performing configuration, Cov-local, improves aggregate F1 over outcome-only training by 3.09 points across seven benchmarks, with a 4.81-point improvement on multi-hop tasks.

Critical strengths
  1. Clear and verifiable supervision. The annotations preserve evidence provenance and dependencies, enabling comparisons between retrieval signals on the same questions and grounding. Retrieval rewards are computed directly from these annotations, without the additional LLM judge used by methods such as CW-GRPO.

  2. Targeted ablations and consistent improvements. The experiments include scalar/local comparisons, credit permutation, and flat-only/no-flat interventions. Cov-local improves F1 over outcome-only training on all seven benchmarks, and the aggregate improvement is not driven by a single dataset.

Critical weaknesses
  1. Cov-local may penalize necessary earlier retrievals. For a conjunctive node, coverage increases only after all required passages have been retrieved, and the resulting local credit goes to the query that completes the node. An earlier query retrieving a necessary passage may therefore register zero progress and receive a negative local retrieval-credit residual after group centering.

  2. The source of the local formulation’s advantage over scalar rewards is not fully isolated. Scalar and local incorporation differ in normalization, feedback granularity, weighting, and token placement. The results do not isolate the contribution of query-local placement. A control that independently normalizes outcome and retrieval signals while retaining trajectory-level credit would help separate these effects.

Questions for authors
  1. Among examples where completing a node requires multiple retrieval steps, what proportion of necessary earlier queries receive a negative local retrieval-credit residual?

  2. What motivated choosing five rollouts per question rather than power-of-two alternatives such as four or eight? Is there a particular methodological or empirical rationale for this choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.