- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes IG-Search, a reinforcement learning framework for search-augmented reasoning. It evaluates each search step through the difference in gold-answer likelihood between real and randomly substituted retrieval contexts, then assigns the resulting signal to the corresponding query tokens within GRPO.
Critical strengths
- The motivation is clear: trajectory-level rewards cannot adequately distinguish the contributions of different searches within the same rollout.
- The framework provides step-level supervision without intermediate annotations or a separately trained reward model.
- The experimental evaluation is fairly comprehensive. In particular, the shuffled and constant reward controls support the importance of aligning the signal with individual search steps.
Critical weaknesses
-
Limited methodological novelty. IGPO already uses changes in gold-answer likelihood to provide dense rewards, while InfoGain-RAG evaluates document contributions through answer-generation confidence. The main distinction here appears to be the combination of random-document counterfactuals and query-token advantage modulation. Although this design is reasonable, the authors should more clearly explain what it enables beyond these existing ideas.
-
The credit assignment is not fully convincing. Equation (2) replaces both the retrieved documents and the model-generated refinement. The resulting signal therefore reflects their combined quality, rather than query quality alone. For example, a useful retrieval followed by an inaccurate refinement could receive a low reward. Assigning this signal entirely to query tokens needs stronger justification.
-
Useful intermediate searches may be under-rewarded. A search that identifies a necessary bridge entity may not immediately increase the probability of the final answer. Its reward could consequently be removed by the dead-zone filter. Please quantify this behavior on questions requiring multiple dependent searches.
-
The all-failure argument needs clarification. Since the trajectory reward includes both answer F1 and a retrieval-specific reward, all final answers being incorrect does not necessarily imply identical total rewards or zero advantages.
-
Some improvements over the strongest baselines are modest. On the larger newer backbones, the average advantage over GiGPO is approximately 0.4–0.5 percentage points. The multi-seed comparison with AutoRefine is useful, but repeated runs against GiGPO and the matched IGPO implementation would better establish whether these smaller gains are reliable.
Questions for authors
See weakness
Thank you for the detailed response. You have addressed most of my concerns, and I will raise my score accordingly.
In particular, grouping trajectories by their total number of searches allows positions within each group to be compared using the same set of trajectories. This addresses my concern about changes in sample composition due to early stopping.
I also appreciate your clarification that a bridge search can provide a key lead for a later query even when it has little immediate effect on the likelihood of the gold answer. I would welcome the inclusion of this analysis and clarification in the revised paper.