- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Explorer-R1, a framework for iterative retrieval that optimizes evidence-state transitions rather than complete retrieval-and-answer trajectories. It introduces teacher-generated transition supervision, followed by SFT and GRPO with a retrieval-grounded reward that jointly evaluates evidence selection, query generation, and termination. Experiments on three multi-hop QA benchmarks demonstrate competitive effectiveness and improved RL training efficiency.
Critical strengths
- I particularly appreciate the Gated Transition Reward design, which prioritizes evidence preservation before rewarding subsequent retrieval and termination decisions. This provides an intuitive and well-motivated training signal for iterative retrieval.
- The method achieves promising results across multiple multi-hop QA benchmarks, while its decoupling of retrieval from answer generation also offers practical flexibility.
Critical weaknesses
- The Gated Transition Reward is not analyzed in sufficient depth. Although the paper includes several overall ablations, the contributions of its individual reward components remain unclear.
- The claim that most existing approaches rely on end-to-end optimization with final-answer rewards seems too broad. Prior work on agentic RAG and tool-integrated reasoning, which also encompasses search and retrieval tools, has explored step-wise rewards (e.g., StepSearch [1] and ToolRL [2]). A clearer distinction would better highlight the specific contribution of Explorer-R1.
[1] Wang, Ziliang, et al. "Stepsearch: Igniting llms search ability via step-wise proximal policy optimization." arXiv preprint arXiv:2505.15107 (2025).
[2] Qian, Cheng, et al. "Toolrl: Reward is all tool learning needs." Advances in Neural Information Processing Systems 38 (2026): 105523-105553.
Questions for authors
- How much does each component of the Gated Transition Reward contribute to the final performance?
- Could you provide sensitivity analyses of the reward weights, particularly the FN/FP penalty ratio?
5 comments
AW2: Positioning against Prior Step-wise Reward Methods
We agree that prior work has explored step-wise optimization for search and tool use. StepSearch provides step-wise PPO rewards based on gold-support gain and retrieval redundancy, while ToolRL evaluates tool selection and argument generation through structured tool-use rewards. These works have made important contributions by moving beyond sparse final-answer supervision toward denser feedback for individual decisions.
However, their reward evaluations primarily focus on the quality of individual decisions against predefined targets or criteria, rather than explicitly modeling how the underlying task state evolves across successive steps. In multi-hop retrieval, this distinction is important because evidence is discovered progressively, and the relevance of an intermediate retrieval decision depends on the evidence accumulated so far. Complete gold evidence is not necessarily available or identifiable at the beginning of exploration. Evaluating every step against a static target or criterion therefore does not fully capture the dependencies between intermediate states and subsequent evidence discovery.
In contrast, ExR1 explicitly optimizes the progression of evidence states by evaluating each transition against the appropriate next-step references conditioned on the current state. Rather than applying a uniform gold-evidence target to every step, ExR1 assesses whether the selected evidence and retrieval actions constitute a valid and productive next step given the evidence accumulated so far. This reflects the progressive nature of multi-hop problem solving, where intermediate steps may first uncover bridging information before revealing the evidence needed to answer the question. Overall, ExR1 moves beyond assigning dense rewards to individual decisions toward optimizing state transitions that follow the evolving logic of multi-hop retrieval.
Changes in the Revised Manuscript
We incorporated all the additional experiments and clarifications into the revised manuscript, as detailed below:
- Section 2, Reinforcement Learning for Retrieval (Lines 123–136): Expanded the discussion of StepSearch and ToolRL, clarifying the distinction between their step-wise reward evaluations and ExR1's optimization of evolving evidence-state transitions.
- Section 4.4, Component Contributions (Lines 433–485): Expanded the reward-component analysis and discussion of their individual contributions to evidence preservation, retrieval supervision, and termination control across three datasets.
- Section 4.4, Table 5 (Lines 487–496): Added positive-advantage coverage analysis to examine how individual reward components influence the optimization signals assigned to different types of correct decisions.
- Appendix B.6, Figure 3 (Lines 1026–1039): Added FN/FP penalty-ratio sensitivity experiments, evaluating final evidence recall and precision under different reward-weight configurations across all three benchmarks.
Thank you again for your detailed feedback. As the discussion phase is approaching its end, we wanted to kindly ask whether you have had an opportunity to review our rebuttal.
We extended the reward-component analysis with positive-advantage coverage over 29,264 sampled actions from 3,658 GRPO groups, providing further insight into how individual components guide different types of correct decisions. We also added FN/FP penalty-ratio sensitivity experiments across all three benchmarks. Furthermore, we clarified the distinction between ExR1 and prior step-wise reward methods such as StepSearch and ToolRL, emphasizing how ExR1 evaluates productive next-step transitions conditioned on the evolving evidence state rather than applying a uniform gold-evidence target throughout retrieval. These additions directly address the concerns regarding reward-component contributions, hyperparameter robustness, and methodological novelty.
We would be grateful if you could let us know whether these results and clarifications resolve your concerns or whether any issues remain. Thank you for your time and consideration.
Trained-policy ablations. We have conducted ablations of the main reward components, including the evidence gate, qD-Pair score, and Oracle reference in Table 4. To further disentangle their contributions, we extended the ablation study to the finer-grained anchor and stop components, and evaluated all configurations on three datasets using BM25. The following table shows that the full reward configuration achieves consistently strong evidence retrieval across all three datasets. The ablations reveal complementary contributions from individual components, with their effects reflected in both evidence quality and downstream QA performance.
Positive-advantage coverage. To further investigate how these components influence the optimization process, we analyzed the positive advantages assigned to reference-correct sampled actions. We examined 29,264 candidate actions from 3,658 logged GRPO groups, categorizing them into retrieval, compression, termination decisions, and jointly correct actions. The following table shows that individual components promote positive reinforcement of correct actions along different decision dimensions. The evidence gate and anchor score contribute to the coverage of correct compression actions, while qD-Pair provides discriminative supervision for retrieval decisions. The Oracle reference supports alternative valid transitions, and the stop score further shapes termination behavior. Importantly, removing individual components changes the distribution of positively reinforced actions, sometimes benefiting one decision category at the expense of others. The complete reward configuration considers these complementary optimization objectives together, providing more comprehensive transition-level supervision rather than favoring only a subset of correct decisions.
Thanks for the suggestion. We conducted additional sensitivity experiments by fixing the FP weight at 1 and varying the FN weight across 1, 2, 4, and 8. Each configuration used the same six-hour training budget. The following table reports final evidence recall and precision under different penalty ratios. Overall, final evidence recall and precision remain stable across all evaluated penalty ratios and datasets. These results demonstrate that ExR1 does not rely on precise weight tuning, and is robust to a broad range of FN/FP configurations.