- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how intermediate retrieval rewards affect RL training for search agents. The authors construct multi-hop QA data with annotated supporting evidence and compare three reward signals: evidence coverage (Cov), dependency-aware coverage (Cov-Dep), and answer matching (AM). They also investigate two ways of incorporating these signals into training: adding them to the trajectory-level reward or assigning credit directly to individual search actions. Experiments on seven QA benchmarks show that intermediate supervision improves performance, with local credit assignment generally working better than scalar rewards. The paper also examines action-reward alignment and how intermediate supervision affects evidence acquisition.
Critical strengths
-
The paper provides a systematic comparison of different intermediate reward signals and credit assignment strategies. The controlled experimental setup makes it easier to understand how these design choices affect search-agent training.
-
I find the analysis of action-level credit assignment interesting. In particular, the reward permutation experiment provides fairly convincing evidence that assigning retrieval credit to the corresponding search action matters, especially for multi-hop questions.
Critical weaknesses
-
Some of the insights are supported by rather small performance differences. For example, Cov-Dep-scalar achieves 52.82 average F1, compared with 52.64 for Cov-scalar. Without statistical significance tests or repeated runs, I am not convinced that these differences are meaningful enough to support conclusions about the reward design.
-
The comparison between scalar rewards and local credit assignment could be better controlled. The two approaches also differ in how the additional rewards are normalized and incorporated into the policy update. It would be helpful to compare them under similar update magnitudes to better isolate the effect of credit assignment.
-
All experiments use a single model backbone and a fixed retrieval setup. The training data also relies on constructed questions with annotated evidence. I would like to see whether the findings hold with other model sizes or more realistic search environments.
Questions for authors
See Weaknesses