acceptodds
Desk rejected at ICLR 2027

IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning has emerged as an effective paradigm for training large language models to perform search-augmented reasoning. However, existing approaches rely on trajectory-level rewards that cannot distinguish precise search queries from vague or redundant ones within a rollout group, and yield a near-zero advantage signal when every sampled trajectory fails. In this paper, we propose IG-Search, a reinforcement learning framework that introduces a step-level reward based on Information Gain (IG). For each search step, IG measures how much the retrieved documents improve the model's confidence in the gold answer relative to a counterfactual baseline of random documents, thereby reflecting the effectiveness of the underlying search query. This signal is fed back to the corresponding search-query tokens via per-token advantage modulation in GRPO, enabling fine-grained, step-level credit assignment within a rollout. Unlike prior step-level methods that require either externally annotated intermediate supervision or shared environment states across trajectories, IG-Search derives its signals from the policy's own generation probabilities, requiring no intermediate annotations beyond standard question-answer pairs. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate that IG-Search achieves an average EM of 0.430 with Qwen2.5-3B, outperforming the strongest trajectory-level baseline MR-Search by 1.6 points and the step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Despite introducing a dense step-level signal, IG-Search adds only about 6.7% to per-step training wall-clock time over the trajectory-level baseline and leaves inference latency unchanged, while still providing a meaningful learning signal even when every sampled trajectory answers incorrectly.

Primary area:
reinforcement learning
Keywords:
information gain, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zgvLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes IG-Search, a reinforcement learning framework for search-augmented reasoning. It evaluates each search step through the difference in gold-answer likelihood between real and randomly substituted retrieval contexts, then assigns the resulting signal to the corresponding query tokens within GRPO.

Critical strengths
  1. The motivation is clear: trajectory-level rewards cannot adequately distinguish the contributions of different searches within the same rollout.
  2. The framework provides step-level supervision without intermediate annotations or a separately trained reward model.
  3. The experimental evaluation is fairly comprehensive. In particular, the shuffled and constant reward controls support the importance of aligning the signal with individual search steps.
Critical weaknesses
  1. Limited methodological novelty. IGPO already uses changes in gold-answer likelihood to provide dense rewards, while InfoGain-RAG evaluates document contributions through answer-generation confidence. The main distinction here appears to be the combination of random-document counterfactuals and query-token advantage modulation. Although this design is reasonable, the authors should more clearly explain what it enables beyond these existing ideas.

  2. The credit assignment is not fully convincing. Equation (2) replaces both the retrieved documents and the model-generated refinement. The resulting signal therefore reflects their combined quality, rather than query quality alone. For example, a useful retrieval followed by an inaccurate refinement could receive a low reward. Assigning this signal entirely to query tokens needs stronger justification.

  3. Useful intermediate searches may be under-rewarded. A search that identifies a necessary bridge entity may not immediately increase the probability of the final answer. Its reward could consequently be removed by the dead-zone filter. Please quantify this behavior on questions requiring multiple dependent searches.

  4. The all-failure argument needs clarification. Since the trajectory reward includes both answer F1 and a retrieval-specific reward, all final answers being incorrect does not necessarily imply identical total rewards or zero advantages.

  5. Some improvements over the strongest baselines are modest. On the larger newer backbones, the average advantage over GiGPO is approximately 0.4–0.5 percentage points. The multi-seed comparison with AutoRefine is useful, but repeated runs against GiGPO and the matched IGPO implementation would better establish whether these smaller gains are reliable.

Questions for authors

See weakness

Reviewer_usnNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Summary:

This paper proposes IG-Search to address the limitation that trajectory-level rewards in search-augmented reasoning struggle to evaluate the contribution of individual search steps. The method defines a step-level reward based on the improvement in the likelihood of the gold answer conditioned on the actual retrieved documents relative to randomly replaced counterfactual contexts. It achieves localized credit assignment via per-token advantage modulation on the search query tokens, while incorporating filtering, scaling, and normalization mechanisms to stabilize training. Experimental results demonstrate that this approach improves QA accuracy and enhances search behaviors in RAG question-answering settings.

Critical strengths

Strengths:

  1. The paper proposes a clear and practical improvement for step-level credit assignment in search-augmented reasoning. The design effectively preserves a meaningful local learning signal even in the all-failure scenario where all final answers are incorrect.
  2. The experiments and ablation studies are comprehensive, providing multi-faceted support for the effectiveness of the method. In addition to evaluating on seven QA benchmarks, the paper includes a controlled comparison with IGPO under a matched implementation, scaling experiments across different model sizes and newer model backbones, as well as thorough ablations on counterfactual baseline designs, advantage modulation scopes, and stabilization mechanisms.
Critical weaknesses

Weaknesses:

  1. Limited novelty. Process rewards based on gold-answer likelihood have precedents such as IGPO, and counterfactual credit assignment is a classic idea in works like Difference Rewards and COMA. The incremental value of this paper lies primarily in the random-document counterfactual replacement and query-token-localized advantage modulation.
  2. Immediate IG may under-credit bridge queries with delayed utility. Even if a search query is necessary for subsequent success, it may not receive sufficient localized credit as long as it leaves the current gold-answer probability unchanged, although the trajectory-level reward remains active. The authors acknowledge this limitation in Appendix V, but the reference to the relevant failure analysis remains a broken placeholder ("Appendix ??"), lacking the corresponding empirical evidence.
  3. The per-position decomposition in Appendix O may be confounded by both delayed utility and sample selection bias. Immediate IG may not adequately evaluate exploratory or bridge steps, while different search positions may consist of different subsets of trajectories: trajectories that acquire answer-bearing evidence early terminate sooner, whereas those still requiring exploration proceed to later steps and enter subsequent statistics. The single-hop vs. multi-hop breakdown in Appendix P partially mitigates task-type confounding, but it does not explicitly control for trajectory composition and stopping behaviors. Therefore, the decreasing mean IG across positions is insufficient to prove that it accurately captures the marginal informational contribution of each step.
Questions for authors

Questions:

  1. Why is the joint utility of documents and refine used solely to modulate the advantage of the search query tokens? The counterfactual construction holds the query fixed while replacing both the documents and refine blocks; thus, IG may also reflect the quality of refinement. However, the existing all-token modulation ablation does not isolate the specific role of refine. Under an identical reward budget, could you provide a crossed comparison evaluating "whether refine is replaced in the counterfactual" versus "whether additional credit is assigned to refine tokens"?
  2. Could query length normalization still introduce a length bias? For a given IG value, each token in a shorter query receives stronger reinforcement or penalization. Keeping the sum of scalar advantages invariant to query length does not guarantee that the actual policy gradients are length-agnostic. Could you analyze this in conjunction with the loss aggregation scheme and conduct length-grouped experiments to examine whether there is a risk of "positive rewards favoring shorter queries and negative rewards favoring longer queries"?
2 comments

Thank you for the detailed response. You have addressed most of my concerns, and I will raise my score accordingly.

In particular, grouping trajectories by their total number of searches allows positions within each group to be compared using the same set of trajectories. This addresses my concern about changes in sample composition due to early stopping.

I also appreciate your clarification that a bridge search can provide a key lead for a later query even when it has little immediate effect on the likelihood of the gold answer. I would welcome the inclusion of this analysis and clarification in the revised paper.

Thank you for your thoughtful feedback and for reconsidering your score. We will incorporate your valuable suggestions, including the additional analysis and clarifications discussed in our response, into the revised paper to further improve its quality and clarity.

Reviewer_iodXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a step-level reward model for search agents based on the results returned by the retrieval model without requiring an external annotator. The proposed solution is an RL framework that scores each search step using a pointwise Information Gain (IG) estimate and uses this score to modulate query-token advantages. With a fixed retriever, the quality of a search query is reflected in the evidence it retrieves, so the informational value of the retrieved documents provides a proxy for query quality. The method is evaluated on seven single- and multi-hop QA benchmarks.

Critical strengths
  • The results on QWEN 3B are encouraging.
  • The proposed solution does not require an external annotator which is often needed in other methods.
  • Experiments are extensive and consistent with prior work on search agents.
Critical weaknesses
  • The results on QWEN 7B is not as strong and the improvements are minor to the extent that I am not sure they are statistically significant. No statistical significance test is done.
  • Related to the last point: Given that the only successful results presented in the paper are for Qwen 2.5 3B, the generalizability of the methods is questionable.
  • Some parts of the paper (including the introduction) look very much LLM generated (example: "The core idea is simple: with a fixed retriever, the quality of a search query is reflected in the evidence it retrieves, so the informational value of the retrieved documents provides a retriever-consistent proxy for query quality.") I am surprised that the paper states at the end that no AI was used at all. (note: I did not use this point in scoring this paper, it's just a minor comment)
Questions for authors

None.

Comments on the paper

3 comments
~Xiyuan_Kai1OpenReview ↗

Concern about the GiGPO comparison.

The GiGPO results appear to be taken directly from its original paper, which uses a maximum of 4 turns. IG-Search allows up to 5 search calls, and this difference in search budget may matter for multi-hop QA. Given the small reported average gains over GiGPO, 0.9 points at 3B and 0.7 at 7B, could the authors clarify the comparison or report results under matched search budgets?

Thank you for your interest in our paper. Appendix T compares IG-Search and GiGPO on Qwen3 and Qwen3.5. For each backbone, both methods use the same training data, retrieval setup, and evaluation protocol, with a maximum of 5 search calls per rollout. We also discussed these comparisons in our response to Reviewer iodX.

Under this shared budget, IG-Search improves average EM over the 7 benchmarks by 0.4–1.1 pp across the 4 backbones. On the 4 multi-hop benchmarks, IG-Search improves average EM by about 1.6 pp on Qwen3-4B and 2.2 pp on Qwen3.5-4B. The two methods have similar multi-hop averages on Qwen3-8B and Qwen3.5-9B.

Thank you for the clarification. To avoid potential confusion, it may be helpful in a future version to ensure that the baseline comparisons in the main table use matched experimental settings, as in Appendix T.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.