acceptodds
Under review as a conference paper at ICLR 2027

Explorer-R1: Efficient Iterative Retrieval via Stepwise Evidence-State Rewarding

Abstract

Iterative retrieval enables large language models to gather evidence step by step, which a single search cannot provide. Existing iterative retrieval methods have used reinforcement learning (RL) to improve multi-hop RAG retrieval for question reasoning and answering. However, most approaches are end-to-end optimizations that use the final answer as the reward signal, which misses intermediate reasoning rewards and makes efficient training difficult; moreover, the model cannot serve as an independent retrieval module for other models. To solve these issues, we propose Explorer-R1 (ExR1), an RL framework for training a plug-and-play multi-hop RAG retrieval model. It reformulates multi-hop retrieval as a sequential decision process. Specifically, we decompose the original end-to-end process into a sequence of state transitions. By tracking evidence state transitions, we collect data to optimize subsequent model decisions. During the RL stage, we design a dense state-level reward that enforces format legality, evidence pruning, and search-result alignment to maintain a compact evidence state. ExR1 decouples retrieval from answering, enabling it to serve as an independent, robust retrieval module that adapts seamlessly and flexibly to downstream collaborative tasks with other large models. Extensive experiments on HotpotQA, MuSiQue, and 2WikiMultihopQA show that ExR1 outperforms existing iterative RAG baselines in recall, converges faster during training, and can effectively collaborate with different answer models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-hop Retrieval, Retrieval-Augmented Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xpQYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Explorer-R1, a framework for iterative retrieval that optimizes evidence-state transitions rather than complete retrieval-and-answer trajectories. It introduces teacher-generated transition supervision, followed by SFT and GRPO with a retrieval-grounded reward that jointly evaluates evidence selection, query generation, and termination. Experiments on three multi-hop QA benchmarks demonstrate competitive effectiveness and improved RL training efficiency.

Critical strengths
  1. I particularly appreciate the Gated Transition Reward design, which prioritizes evidence preservation before rewarding subsequent retrieval and termination decisions. This provides an intuitive and well-motivated training signal for iterative retrieval.
  2. The method achieves promising results across multiple multi-hop QA benchmarks, while its decoupling of retrieval from answer generation also offers practical flexibility.
Critical weaknesses
  1. The Gated Transition Reward is not analyzed in sufficient depth. Although the paper includes several overall ablations, the contributions of its individual reward components remain unclear.
  2. The claim that most existing approaches rely on end-to-end optimization with final-answer rewards seems too broad. Prior work on agentic RAG and tool-integrated reasoning, which also encompasses search and retrieval tools, has explored step-wise rewards (e.g., StepSearch [1] and ToolRL [2]). A clearer distinction would better highlight the specific contribution of Explorer-R1.

[1] Wang, Ziliang, et al. "Stepsearch: Igniting llms search ability via step-wise proximal policy optimization." arXiv preprint arXiv:2505.15107 (2025).

[2] Qian, Cheng, et al. "Toolrl: Reward is all tool learning needs." Advances in Neural Information Processing Systems 38 (2026): 105523-105553.

Questions for authors
  1. How much does each component of the Gated Transition Reward contribute to the final performance?
  2. Could you provide sensitivity analyses of the reward weights, particularly the FN/FP penalty ratio?
5 comments

AW1 & AQ1: Component-wise Analysis of the Gated Transition Reward

Trained-policy ablations. We have conducted ablations of the main reward components, including the evidence gate, qD-Pair score, and Oracle reference in Table 4. To further disentangle their contributions, we extended the ablation study to the finer-grained anchor and stop components, and evaluated all configurations on three datasets using BM25. The following table shows that the full reward configuration achieves consistently strong evidence retrieval across all three datasets. The ablations reveal complementary contributions from individual components, with their effects reflected in both evidence quality and downstream QA performance.

VariantHotpot F1Final R.MuSiQue F1Final R.2Wiki F1Final R.
ExR159.679.526.056.647.382.5
w/o evidence gate58.577.625.555.746.480.9
w/o qD-Pair59.678.825.656.247.481.4
w/o Oracle59.176.124.854.347.481.1
w/o anchor58.677.526.156.946.681.4
w/o stop59.878.326.055.747.080.8

Positive-advantage coverage. To further investigate how these components influence the optimization process, we analyzed the positive advantages assigned to reference-correct sampled actions. We examined 29,264 candidate actions from 3,658 logged GRPO groups, categorizing them into retrieval, compression, termination decisions, and jointly correct actions. The following table shows that individual components promote positive reinforcement of correct actions along different decision dimensions. The evidence gate and anchor score contribute to the coverage of correct compression actions, while qD-Pair provides discriminative supervision for retrieval decisions. The Oracle reference supports alternative valid transitions, and the stop score further shapes termination behavior. Importantly, removing individual components changes the distribution of positively reinforced actions, sometimes benefiting one decision category at the expense of others. The complete reward configuration considers these complementary optimization objectives together, providing more comprehensive transition-level supervision rather than favoring only a subset of correct decisions.

Reward configurationRetrievalCompressionDecisionJoint
ExR12,5275,79012,588755
w/o evidence gate2,5495,45413,003755
w/o qD-Pair4255,4156,385390
w/o Oracle2,5485,63612,522744
w/o anchor2,5615,23412,083756
w/o stop2,5585,75112,587756

AQ2: Sensitivity to the FN/FP Penalty Ratio

Thanks for the suggestion. We conducted additional sensitivity experiments by fixing the FP weight at 1 and varying the FN weight across 1, 2, 4, and 8. Each configuration used the same six-hour training budget. The following table reports final evidence recall and precision under different penalty ratios. Overall, final evidence recall and precision remain stable across all evaluated penalty ratios and datasets. These results demonstrate that ExR1 does not rely on precise weight tuning, and is robust to a broad range of FN/FP configurations.

DatasetFN/FPFinal Recall (%)Final Precision (%)
HotpotQA1:180.1088.37
2:179.2088.50
4:179.5087.82
8:179.0088.97
MuSiQue1:156.6274.66
2:156.9174.92
4:156.5774.42
8:156.8674.47
2WikiMultihopQA1:182.1594.81
2:183.1094.39
4:182.4794.32
8:181.7394.48

AW2: Positioning against Prior Step-wise Reward Methods

We agree that prior work has explored step-wise optimization for search and tool use. StepSearch provides step-wise PPO rewards based on gold-support gain and retrieval redundancy, while ToolRL evaluates tool selection and argument generation through structured tool-use rewards. These works have made important contributions by moving beyond sparse final-answer supervision toward denser feedback for individual decisions.

However, their reward evaluations primarily focus on the quality of individual decisions against predefined targets or criteria, rather than explicitly modeling how the underlying task state evolves across successive steps. In multi-hop retrieval, this distinction is important because evidence is discovered progressively, and the relevance of an intermediate retrieval decision depends on the evidence accumulated so far. Complete gold evidence is not necessarily available or identifiable at the beginning of exploration. Evaluating every step against a static target or criterion therefore does not fully capture the dependencies between intermediate states and subsequent evidence discovery.

In contrast, ExR1 explicitly optimizes the progression of evidence states by evaluating each transition against the appropriate next-step references conditioned on the current state. Rather than applying a uniform gold-evidence target to every step, ExR1 assesses whether the selected evidence and retrieval actions constitute a valid and productive next step given the evidence accumulated so far. This reflects the progressive nature of multi-hop problem solving, where intermediate steps may first uncover bridging information before revealing the evidence needed to answer the question. Overall, ExR1 moves beyond assigning dense rewards to individual decisions toward optimizing state transitions that follow the evolving logic of multi-hop retrieval.

Changes in the Revised Manuscript

We incorporated all the additional experiments and clarifications into the revised manuscript, as detailed below:

  • Section 2, Reinforcement Learning for Retrieval (Lines 123–136): Expanded the discussion of StepSearch and ToolRL, clarifying the distinction between their step-wise reward evaluations and ExR1's optimization of evolving evidence-state transitions.
  • Section 4.4, Component Contributions (Lines 433–485): Expanded the reward-component analysis and discussion of their individual contributions to evidence preservation, retrieval supervision, and termination control across three datasets.
  • Section 4.4, Table 5 (Lines 487–496): Added positive-advantage coverage analysis to examine how individual reward components influence the optimization signals assigned to different types of correct decisions.
  • Appendix B.6, Figure 3 (Lines 1026–1039): Added FN/FP penalty-ratio sensitivity experiments, evaluating final evidence recall and precision under different reward-weight configurations across all three benchmarks.

Thank you again for your detailed feedback. As the discussion phase is approaching its end, we wanted to kindly ask whether you have had an opportunity to review our rebuttal.

We extended the reward-component analysis with positive-advantage coverage over 29,264 sampled actions from 3,658 GRPO groups, providing further insight into how individual components guide different types of correct decisions. We also added FN/FP penalty-ratio sensitivity experiments across all three benchmarks. Furthermore, we clarified the distinction between ExR1 and prior step-wise reward methods such as StepSearch and ToolRL, emphasizing how ExR1 evaluates productive next-step transitions conditioned on the evolving evidence state rather than applying a uniform gold-evidence target throughout retrieval. These additions directly address the concerns regarding reward-component contributions, hyperparameter robustness, and methodological novelty.

We would be grateful if you could let us know whether these results and clarifications resolve your concerns or whether any issues remain. Thank you for your time and consideration.

Reviewer_P6DmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Explorer-R1 (ExR1), a framework that trains a standalone multi-hop retriever through stepwise evidence-state optimization. Instead of optimizing complete retrieval trajectories using final-answer rewards, ExR1 applies GRPO to individual state transitions, with rewards for evidence selection, query matching, and termination decisions. The framework uses teacher-generated data for SFT and RL, and separates retrieval from answer generation. Experiments on three multi-hop QA benchmarks show improvements in retrieval quality, training efficiency, and collaboration with different answer models.

Critical strengths
  1. The transition-level optimization provides a practical alternative to end-to-end RL for iterative retrieval. It enables more direct supervision of intermediate decisions without requiring complete search-and-answer rollouts.
  2. The experimental results are promising. ExR1 outperforms several existing iterative retrieval methods across different retrieval backends. Its ability to work with multiple frozen answer models supports the value of separating retrieval from answer generation.
Critical weaknesses
  1. The baseline comparisons are insufficient. Several recent and relevant methods are missing, particularly SPARKLE and PAR²-RAG, which are already cited in the paper. Comparisons with stronger plug-and-play retrieval methods and training-free approaches would better establish the advantages of ExR1.
  2. The experimental validation is limited in scope. All trained explorers use the same 1.5B backbone, and the reward-component ablations are conducted only on MuSiQue. The scalability of the method and the consistency of these components across datasets remain unclear.
Questions for authors
  1. Please consider adding comparisons with SPARKLE and recent training-free retrieval methods such as PAR²-RAG, particularly in the plug-and-play evaluation.
  2. How does ExR1 perform with larger explorer models, such as 3B or 7B? It would also be useful to extend the reward-component ablations to HotpotQA and 2WikiMultiHopQA.
4 comments

AW1 & AQ1: Additional Retrieval Baselines and Plug-and-Play Evaluation

Thank you for the suggestion. We expanded the baseline comparison with PAR²-RAG and SPARKLE, covering both prompt-based retrieval and learned retrieval policies. PAR²-RAG was implemented following the published algorithm description. For SPARKLE, we followed its public implementation. Since its original SFT dataset was unavailable, we converted our teacher-generated records into the required format using Qwen3.5-Plus, followed by SFT and 36 hours of PPO training. Both baselines use 1.5B models. All methods were evaluated with frozen BM25 and DPR indexes.

Main Baseline Comparison. The following table reports QA F1 across three benchmarks. ExR1 outperforms both added baselines across all six dataset–retriever combinations. This advantage also persists against SPARKLE, which uses teacher-supervised initialization followed by RL.

BackendMethodHotpotQAMuSiQue2Wiki
BM25PAR²-RAG37.712.719.6
SPARKLE36.914.333.2
ExR161.126.046.6
DPRPAR²-RAG36.912.623.0
SPARKLE37.317.632.1
ExR158.327.946.5

Comparative Training Progress. We further evaluated SPARKLE`s training trajectories on HotpotQA with BM25 under a 36-hour budget. The following table shows that SPARKLE remains close to its SFT performance, whereas ExR1 continues to improve retrieval recall.

Training budgetSPARKLE F1SPARKLE R@5ExR1 F1ExR1 R@5
SFT36.9475.1056.6777.35
6h36.5675.2561.1479.60
12h36.8575.0560.0379.85
24h36.8775.1560.4980.20
36h36.8474.7059.5180.40

Plug-and-play Evaluation. To directly evaluate the added baselines in downstream collaboration, we used their retrieved evidence with three frozen answer models: Qwen2.5-7B-Instruct, Qwen3.5-Plus, and GPT-4o-mini. Each entry reports QA F1 and answerer-side token consumption (F1 / Tokens). The following table shows that ExR1 achieves the highest QA F1 across all nine dataset–answerer combinations while consistently requiring fewer answerer-side tokens. Its compact evidence states substantially reduce downstream context consumption compared with both PAR²-RAG and SPARKLE, demonstrating a favorable combination of answer quality and inference efficiency. These results demonstrate that ExR1 provides more effective and token-efficient evidence for diverse frozen answer models, consistently improving downstream QA performance.

DatasetMethodQwen2.5-7BQwen3.5-PlusGPT-4o-mini
HotpotQAPAR²-RAG54.3 / 237064.5 / 237663.8 / 2236
SPARKLE56.2 / 87153.2 / 88957.6 / 818
ExR169.9 / 34869.0 / 45769.4 / 413
MuSiQuePAR²-RAG15.4 / 273421.6 / 274527.1 / 2586
SPARKLE21.7 / 92921.8 / 94627.2 / 869
ExR126.8 / 50627.3 / 52131.1 / 467
2WikiPAR²-RAG32.3 / 296736.2 / 296136.4 / 2780
SPARKLE43.2 / 89741.9 / 90445.4 / 838
ExR153.9 / 56054.0 / 56758.1 / 521

AW2 & AQ2: Explorer Scaling and Cross-Dataset Reward Ablations

Thank you for pointing out the limited model-scale evaluation and the lack of cross-dataset ablations. We conducted additional experiments to address both concerns.

Explorer Scaling. We extended the explorer backbone from 1.5B to 3B parameters and evaluated it with both BM25 and DPR. The following table shows that scaling the explorer from 1.5B to 3B consistently improves both QA F1 and retrieval Recall@3 across all three datasets and both retrieval backends, with the answerer fixed at 1.5B.

BackendDatasetExR1 1.5B (F1 / R@3)ExR1 3B (QA1.5B) (F1 / R@3)
BM25HotpotQA61.1 / 79.662.7 / 83.9
MuSiQue26.0 / 52.231.1 / 59.8
2Wiki46.6 / 81.448.2 / 83.0
DPRHotpotQA58.3 / 78.760.5 / 81.2
MuSiQue27.9 / 54.632.1 / 61.0
2Wiki46.5 / 80.349.6 / 82.8

Cross-dataset reward ablations. We extended the original MuSiQue-only reward ablation to HotpotQA and 2WikiMultihopQA. The following table shows that the expanded evaluations confirm the original findings on MuSiQue. Each component provides complementary signals. The results support the generalization of the proposed reward design across all three multi-hop QA benchmarks.

VariantHotpot F1Hotpot Final R.MuSiQue F1MuSiQue Final R.2Wiki F12Wiki Final R.
ExR159.679.526.056.647.382.5
w/o evidence gate58.577.625.555.746.480.9
w/o qD-Pair59.678.825.656.247.481.4
w/o Oracle59.176.124.854.347.481.1

Changes in the Revised Manuscript

We incorporated all the additional experiments and clarifications into the revised manuscript, as detailed below:

  • Section 4.1, Models and Baselines (Lines 318–323), and Table 1 (Lines 325–354): Added PAR²-RAG and SPARKLE to the main comparisons under BM25 and DPR, and included the 3B Explorer results.
  • Section 4.2, Evidence Flow and Inference Cost (Lines 379–387): Added SPARKLE to the evidence-flow and inference-cost comparison, including final evidence recall, precision, document count, and input-token consumption.
  • Section 4.3, Convergence and Training Efficiency (Lines 389–399, 423–457): Added SPARKLE's training trajectory and the 3B Explorer training evaluations, including full-parameter and LoRA variants.
  • Section 4.4, Component Contributions (Lines 433–451): Extended the reward-component ablations from MuSiQue to HotpotQA and 2WikiMultihopQA.
  • Section 4.5, Plug-and-Play Collaboration (Lines 498–528): Added PAR²-RAG and SPARKLE to the evaluation with three frozen downstream answer models, reporting both QA performance and answerer-side token consumption.

Thank you again for your detailed feedback. As the discussion phase is approaching its end, we wanted to kindly ask whether you have had an opportunity to review our rebuttal.

We added PAR²-RAG and SPARKLE to the main benchmark comparisons under BM25 and DPR, and further evaluated their retrieved evidence with three frozen answer models across all three datasets. We also included evidence-flow, inference-cost, and training-trajectory comparisons with SPARKLE. In addition, we extended the explorer to 3B and expanded reward-component ablations to HotpotQA and 2WikiMultihopQA. These additions directly address the concerns regarding baseline coverage, plug-and-play effectiveness, model scalability, and cross-dataset validation.

We would be grateful if you could let us know whether these results and clarifications resolve your concerns or whether any issues remain. Thank you for your time and consideration.

Dear Reviewer P6Dm,

As today is the final day of the author–reviewer discussion period, we would be grateful if you could review our submitted rebuttal before the discussion closes. Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_s5fbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an efficient sequential decision making framework for multi-hop question answering with a 1.5B distilled retriever with reinforcement learning. In the RL framework, the designs necessary reward signals based on evidence cost, query similarity and hallucination mitigation; and provides their justification in the ablation section. The work demonstrates the efficiency of the proposed framework in terms of low-rank optimization, GPU time and tokens. Overall, the method improves over existing methods in terms of efficiency and performance in multiple knowledge-intensive tasks.

Critical strengths

Strong Reward Signals: The method curates strong reward signal that is diverse and effective in fine-tuning the 1.5B model. It includes the evidence cost, query awareness, gating and oracle reference. This strong signal drives the performance as shown in Table 4 (e.g., without oracle reference ExR1 EM drop to 17.4 to 18.5).

Composability: The resulting model can be used standalone with much larger model for final answer generation (e.g., 69.4 F1 with GTP-4o-mini). This can be useful for modern multi-stage RAG frameworks, where multiple models are chained to improve performance.

Critical weaknesses

Scalability: The method only demonstrate improvement in 1.5B settings. It is important for these methods to scale for proving them to be practically useful, for example, in case of Search-R1, the gaps between 1.5B and 3B models are significantly large.

Distillation: The model uses a strong supervisory signal from latest large model, like Qwen3.5-Plus. It is unclear how this might affected the methodological gains with the baselines which might not require such kind of distillation.

Questions for authors
  1. Does the framework scale to larger settings that 1.5B?
  2. How do we isolate the contribution of external distillation vs methodological contribution for fair comparison with baselines?
4 comments

AW1 & AQ1: Scalability Beyond the 1.5B Setting

Thanks for the suggestion. We extended our experiments to the Qwen2.5-3B setting to examine model scaling and the effectiveness of transition-level optimization at a larger scale.

Model Scaling and RL Effectiveness. We evaluated 1.5B and 3B explorers at the SFT stage, followed by GRPO optimization of the 3B explorer, using the same three benchmarks and a frozen 1.5B answerer. The following table shows consistent improvements in both answer quality and final evidence recall when scaling from 1.5B to 3B. Applying GRPO further improves the 3B explorer across all three datasets, demonstrating that transition-level optimization remains effective beyond the original model scale.

Dataset1.5B SFT F13B SFT F13B GRPO F11.5B SFT Final Recall3B SFT Final Recall3B GRPO Final Recall
HotpotQA56.6759.4862.7377.3579.6083.90
MuSiQue25.0628.8531.1054.5357.1759.93
2Wiki44.4547.4448.2479.6382.6884.55

Comparison with Larger Baselines. We further compared ExR1 with Search-R1 at 3B under both BM25 and DPR. The following table shows that with the answerer fixed at 1.5B, ExR1 consistently outperforms Search-R1 across all six settings and using a 3B answerer further improves ExR1's performance.

BackendDatasetExR1 3B (QA1.5B)ExR1 3B (QA3B)Search-R1 3B
BM25HotpotQA62.768.253.9
BM25MuSiQue31.138.422.5
BM252Wiki48.262.147.1
DPRHotpotQA60.566.449.3
DPRMuSiQue32.138.525.7
DPR2Wiki49.661.147.2

AW2 & AQ2: Distillation and Methodological Contributions

We agree that teacher-generated evidence-state trajectories are an important supervision in ExR1. However, ExR1 is not merely a distillation-based SFT approach that imitates teacher actions. Its key contribution is to further optimize the student's evidence-state transition policy through retrieval-grounded RL. After SFT, the student independently samples candidate actions at each evidence state, and GRPO optimizes them according to evidence preservation, retrieval outcomes, and termination decisions.

Comparison with Teacher-supervised Baselines. We compared ExR1 with RAG-R1-SQ and RAG-R1-MQ, which also use teacher trajectories for SFT, and SPARKLE. The following table shows that ExR1 consistently outperforms these methods across both retrieval backends and all three datasets, indicating that its advantage is not simply due to the availability of teacher-generated supervision.

BackendDatasetExR1RAG-R1-SQRAG-R1-MQSPARKLE
BM25HotpotQA61.149.847.536.9
BM25MuSiQue26.021.218.014.3
BM252Wiki46.643.944.333.2
DPRHotpotQA58.346.147.937.3
DPRMuSiQue27.923.318.317.6
DPR2Wiki46.540.541.332.1

Optimization Trajectories and Performance Gains. More importantly, we examined whether different methods could improve beyond their teacher-supervised initialization. For a more controlled comparison, we trained SPARKLE using data converted from ExR1's original teacher-generated trajectories with the same teacher model and the same training-time budget. The following table summarizes the initial and final F1, AUC, and rollout-token consumption on HotpotQA. ExR1 consistently improves beyond its SFT initialization, whereas the teacher-supervised baselines exhibit limited or unstable gains. This indicates that ExR1's improvements do not merely come from imitating teacher-generated actions, but from further optimizing the student's transition policy through retrieval-grounded rewards.

MethodOrigin F136h F1AUCRollout Tok.
RAG-R1-SQ49.3048.9748.840.40M
RAG-R1-MQ47.1547.4947.060.40M
SPARKLE36.9436.8436.820.15M
ExR1 (Full)56.6758.3257.370.41M
ExR1 (LoRA)56.6759.5160.001.95M

Changes in the Revised Manuscript

We incorporated all the additional experiments and clarifications into the revised manuscript, as detailed below:

  • Section 4.1, Models and Baselines (Lines 318–323), and Table 1 (Lines 325–354): Added the 3B Explorer evaluation and comparisons with Search-R1 (3B), RAG-R1-SQ, and RAG-R1-MQ, distinguishing explorer scaling from answerer scaling.
  • Section 4.2, Main Effectiveness (Lines 369–403): Added the analysis of 3B Explorer performance with a fixed 1.5B answerer and the additional evaluation with a 3B answerer.
  • Section 4.3, Convergence and Training Efficiency (Lines 389–399, 423–457): Added SPARKLE training trajectories and 3B full-parameter and LoRA training results. Expanded the discussion to distinguish SFT initialization from subsequent GRPO improvements and clarify the role of transition-level optimization.

Thank you again for your detailed feedback. As the discussion phase is approaching its end, we wanted to kindly ask whether you have had an opportunity to review our rebuttal.

We extended ExR1 to the 3B setting and evaluated both SFT initialization and subsequent GRPO optimization across all three benchmarks. We also added comparisons with teacher-supervised baselines, including RAG-R1-SQ, RAG-R1-MQ, and SPARKLE, together with aligned training-trajectory evaluations. These additional experiments examine the scalability of ExR1 and demonstrate that its improvements extend beyond teacher imitation through continued transition-level optimization.

We would be grateful if you could let us know whether these results and clarifications resolve your concerns or whether any issues remain. Thank you for your time and consideration.

Dear Reviewer s5fb,

As today is the final day of the author–reviewer discussion period, we would be grateful if you could review our submitted rebuttal before the discussion closes. Thank you again for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment

Dear Area Chair,

As the discussion period approaches its end, we would like to summarize the main additions addressing the reviewers’ concerns. We posted our detailed responses on November 13 and have incorporated the additional experiments and clarifications into the revised manuscript.

1. Additional baselines and plug-and-play evaluation. To address Reviewer P6Dm’s concerns, we added SPARKLE and PAR²-RAG under both BM25 and DPR. In the plug-and-play evaluation, ExR1 achieved higher QA F1 than both baselines across all nine combinations of three datasets and three frozen answer models, while requiring fewer answerer-side tokens.

2. Scaling beyond 1.5B. In response to Reviewers s5fb and P6Dm, we extended the explorer to 3B while keeping the answerer fixed at 1.5B. Scaling improved QA F1 and retrieval recall across all three datasets and both retrieval backends. We also added comparisons with Search-R1 at 3B and demonstrated further improvements from SFT to GRPO for the 3B explorer.

3. Contributions beyond teacher-supervised initialization. To address Reviewer s5fb’s concern, we added comparisons with RAG-R1-SQ, RAG-R1-MQ, and SPARKLE, together with SFT-to-RL training trajectories. The SPARKLE comparison used data converted from the same source teacher trajectories and the same training-time budget. The results show gains beyond ExR1’s SFT initialization, supporting the incremental contribution of transition-level RL.

4. Reward analysis and sensitivity. Addressing Reviewers xpQY and P6Dm, we extended reward-component ablations to all three datasets, added finer-grained anchor and stop ablations, and analyzed optimization signals over 29,264 sampled actions. We also conducted FN/FP sensitivity experiments, finding relatively stable evidence recall and precision across the tested configurations.

5. Positioning relative to prior work. In response to Reviewer xpQY, we explicitly acknowledged existing step-wise reward methods and clarified ExR1’s relationship to StepSearch and ToolRL, emphasizing its optimization of evolving evidence-state transitions in a standalone retriever.

Our individual responses provide the detailed results and corresponding locations in the revised manuscript. As we have not yet received follow-up feedback, we would appreciate your assistance in encouraging the reviewers to assess these additions and identify any remaining concerns before the discussion closes. We remain available to provide further clarification promptly.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.