acceptodds
Under review as a conference paper at ICLR 2027

R3PO: Unified Policy Optimization For Agentic RAG

Abstract

Agentic retrieval-augmented generation (RAG) systems depend on a reasoner and a retriever, yet a schism in their optimization—reinforcement learning (RL) for the reasoner versus indirect signals for the retriever—hinders true end-to-end alignment with task outcomes. We introduce Reasoner-Retriever Reinforcement Policy Optimization (R3PO), a staged framework that applies a common, outcome-driven policy optimization interface to both reasoner and retriever actions. Our approach is built on three pillars: principled policy construction, which defines a stochastic policy that respects the geometry of the pretrained retriever's native scoring function; principled adaptation, which uses a unified RL objective to refine both components with trajectory-level rewards; and principled generalization, which confronts the critical “candidate-to-corpus chasm.” We prove that adapting a retriever on a small candidate pool makes it blind to score shifts that are catastrophic for full-corpus retrieval. R3PO solves this by introducing background documents as anchors during training, grounding the learned policy and ensuring it generalizes beyond the candidate set. This preserves the retriever's efficient inner-product structure, enabling direct, full-corpus deployment. Across seven QA benchmarks and three backbones, R3PO consistently achieves higher accuracy with greater efficiency, delivering relative gains of 2.2%-3.5% in average exact match (EM) while even enhancing an already agent-adapted retriever. These results demonstrate that a principled, unified RL approach unlocks significant performance gains.

Primary area:
reinforcement learning
Keywords:
Retrieval-Augmented Generation, Agentic Search, Reinforcement Learning, Dense Retrieval, Policy Optimization, Retriever Post-Training, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_RXrwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

R3PO uses a two-stage GRPO framework to train the reasoner and dense retriever in agentic RAG. It turns the retriever’s native inner-product scores into a document-selection policy and updates that policy using rewards from completed QA trajectories. The paper shows that training on a small candidate pool cannot detect a uniform score shift relative to documents outside the pool, and adds randomly sampled background documents as anchors. Across seven QA datasets and three retriever backbones, the authors report improvements in answer exact match (EM) and retriever Pass@3. An E5 ablation compares candidate-pool reranking with retrieval from a rebuilt full-corpus index.

Critical strengths
  1. The paper identifies a concrete deployment problem. Its candidate-set identifiability analysis explains a specific failure mode and motivates cross-pool comparisons. This is the paper’s most compelling contribution.
  2. The evaluation includes full-corpus retrieval. The authors re-encode the corpus and rebuild the index rather than evaluating only within a fixed candidate pool. Reporting both end-to-end EM and retriever Pass@3 provides initial evidence that the method improves retrieval itself.
Critical weaknesses
  1. The evidence for the central mechanism is incomplete. The key background-document ablation is reported only for E5. The paper does not directly measure the proposed cross-pool score drift or report variation across random seeds. Table 11 is consistent with the theory, but does not establish that anchoring works through the proposed mechanism or justify the claim that it ensures full-corpus generalization. This limits confidence in the paper’s main contribution.
  2. The novelty and “unified end-to-end” claims need narrower framing. Prior work has already applied stochastic document selection and GRPO to dense retrievers [1]. Moreover, R3PO trains its two components sequentially with different rewards, rather than jointly optimizing them. Its distinctive contribution appears to be the cross-pool analysis and anchoring strategy; the current framing makes that contribution harder to assess precisely.

[1] Y. Zhang et al., “Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG,” 2026.

Questions for authors
  1. In the two conditions in Table 11, how do the scores, ranks, or probability mass of the Top-10 documents change relative to documents outside that pool after training? Is there direct evidence of the uniform cross-pool drift described by Theorem 2?
  2. How much probability mass do background documents typically receive in the training softmax? If that mass is very small, how strong is the resulting constraint on cross-pool drift?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RXrw (1/4)

Many thanks for your positive review and valuable feedback! We believe that addressing your comments has resulted in improving the clarity and presentation of the paper's contributions and has brought the paper to a higher standard. Below we address your comments individually in order. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all equations, sections, tables, and bibliographical references refer to those in the revised paper.

1. Strengthening Evidence for the Central Anchoring Mechanism & Stability (W1)

In the original submission, we reported rebuilt full-corpus retrieval gains across all three backbones (E5, Agentic-R, Qwen-0.6B) on end-to-end EM (Table 2) and isolated Pass@3 (Table 16, formerly Table 12), together with the E5 K3=0K_3=0 vs. K3=256K_3=256 reranking-to-retrieval reversal (Table 15, formerly Table 11). You rightly noted that original Table 11 alone did not isolate the anchoring mechanism from rollout sampling, lacked direct cross-pool drift measurements and seed-stability checks, and used overly strong "ensures generalization" wording. In the revised paper, we added:

  • New Anchor-Only ablation isolating loss anchoring (App. D.3, Table 15): During Stage-II rollouts, Anchor-Only samples actions strictly from Top10(q)\mathrm{Top}_{10}(q) (never passing background passages to the reasoner), keeping the 256 random corpus passages only in the softmax denominator and reference KL penalty. Its 47.7% full-corpus EM (vs. 47.8% standard R3PO and 47.0% for K3=0K_3=0, matching or exceeding standard R3PO on all four multi-hop datasets) supports the role of cross-pool anchoring directly through the training loss.
  • New direct cross-pool score, probability-mass, and SciFact diagnostics (§7.2, Table 4): Detailed under Q1 below.
  • Cross-backbone controlled validation (beyond E5) & stability (§7.1–7.3, Tables 2, 3, 5; App. C.3): Rebuilt full-corpus gains hold consistently across all three backbones (E5, Agentic-R, Qwen-0.6B) on original EM (Table 2) and Pass@3 (Table 16) as well as new Token F1, LLM-Judge, and nDCG@3 (Table 3). Crucially, Table 5 (expanded in blue) provides a direct controlled validation on a second backbone (Qwen-0.6B): once HARR is strictly aligned with R3PO's Query only state (45.5%, original) and exact composite reward (HARR + Our Reward, 45.6%, new) under the same G=5G=5 budget, the remaining difference is precisely freezing the document index versus R3PO's background-anchored dual-encoder adaptation and rebuilt full-corpus index—lifting accuracy further to 46.2% EM (46.21% and 46.20% across two independent evaluation seeds; given the ~2,304 GPU-h cost per Stage-II run, these two runs measure multi-turn inference stability across evaluation seeds, while training-level stability is corroborated by consistent gains across all three backbones and the independent Anchor-Only training run in Table 15).
  • Scoped claims: As detailed under W2, we replaced unconditional "ensures generalization" phrasing with empirically scoped statements aligned with Theorem 2 and Appendix B.6–B.8.
AuthorsOpenReview ↗

Response to Reviewer RXrw (2/4)

2. Narrower Framing of "Unified", Sequential Training, and Prior Retriever RL (W2)

We agree completely and revised Abstract, §1, §2, §3, §5, §6, Appendix A, and Appendix E to:

  1. Explicitly state upfront that R3PO is a sequentially staged two-stage framework with stage-specific trajectory rewards (not simultaneous joint optimization), and that "unified" refers strictly to casting both token generation and native document selection under a shared rollout–outcome–GRPO policy interface;
  2. Credit HARR [1] (alongside ROPG-RL and PAO) prominently in §1–§2 and Appendix A for stochastic policy-gradient/GRPO retriever training, so R3PO's distinctive contribution in agentic RAG—adapting both query and document representations via the shared bi-encoder and anchoring cross-pool drift for rebuilt full-corpus deployment rather than freezing the document index—can be assessed cleanly; and
  3. Replace "ensures generalization" with "constrains cross-pool score drift" / "generalizes reliably".
AuthorsOpenReview ↗

Response to Reviewer RXrw (3/4)

3. Score Separation & Probability-Mass Shifts Across the Two Conditions of Original Table 11 (Q1)

Direct Answer: Yes—direct diagnostics show systematic cross-pool drift between Top10(q)\mathrm{Top}_{10}(q) and outside documents when anchors are omitted. Evaluating Frozen E5 (ϕ0\phi_{0}), Unanchored R3PO (K3=0K_{3}=0), and Anchored R3PO (K3=256K_{3}=256) on identical fixed diagnostic mixed pools C1(q)=Top10(q)∪Bg256C_{1}(q) = \mathrm{Top}_{10}(q) \cup \mathrm{Bg}_{256} (§7.2 and Table 4, new) shows that without anchors (K3=0K_3=0), outside background probability mass jumps 4x from 0.0002 to 0.0008 as the Top-10 boundary margin narrows to 0.10, whereas anchoring (K3=256K_3=256) constrains background mass to 0.0003 (6x smaller drift, +0.0001 vs. +0.0006) and maintains a +30% wider boundary margin (0.13):

  • Native cosine boundary margin (§7.2): Theorem 2 proves that a uniform block shift between Top10(q)\mathrm{Top}_{10}(q) and outside documents lies in the exact nullspace of the K3=0K_3=0 objective (App. B.5); in practice, parameterized shared-encoder SGD manifests this unpenalized direction as systematic cross-pool encroachment. Directly measuring E[min⁡d∈Top10(q)sϕ(q,d)−max⁡u∈Bg256sϕ(q,u)]\mathbb{E}[\min_{d\in\mathrm{Top}_{10}(q)}s_\phi(q,d)-\max_{u\in\mathrm{Bg}_{256}}s_\phi(q,u)] shows that anchored training (K3=256K_3=256) maintains a +30% wider mean boundary margin of 0.13 versus 0.10 for unanchored training (K3=0K_3=0).
  • Relative probability-mass shift (Table 4): Under frozen E5 (τ=0.01,T=1\tau=0.01, T=1), the 256 background passages receive a mean total softmax mass E[∑u∈Bg256pϕ(u∣q,C1)]\mathbb{E}[\sum_{u\in\mathrm{Bg}_{256}}p_\phi(u\mid q,C_1)] of 0.0002 (so Top10(q)\mathrm{Top}_{10}(q) holds 0.9998). Training without background anchors (K3=0K_3=0) causes outside-pool mass to drift upward 4x to 0.0008 (+0.0006 drift), whereas background-trained R3PO (K3=256K_3=256) constrains it to 0.0003 (+0.0001 drift, 6x smaller)—and better preserves zero-shot retrieval on out-of-corpus biomedical SciFact (nDCG@10: 0.716 vs. 0.705, compared with 0.719 frozen; Recall@10: 0.835 vs. 0.826, compared with 0.846 frozen).

Cross-pool probability diagnostics and SciFact retention (Table 4).

Configuration (E5)Full-Corpus QA EM (%) ↑\uparrowBg. Pool Mass ↓\downarrowSciFact nDCG@10 ↑\uparrowSciFact Recall@10 ↑\uparrow
Frozen E5 (ϕ0\phi_0)46.50.00020.7190.846
R3PO w/o Background (K3=0K_3=0)47.00.00080.7050.826
R3PO w/ Background (K3=256K_3=256)47.80.00030.7160.835
AuthorsOpenReview ↗

Response to Reviewer RXrw (4/4)

4. Softmax Probability Mass of Background Documents & Strength of the Constraint (Q2)

  • Measured softmax probability mass (Table 4, new): On the evaluated checkpoints and diagnostic mixed pools (τ=0.01,T=1\tau=0.01, T=1), all 256 background documents combined receive a mean total selection probability of 0.0002 under frozen E5 and 0.0003 after anchored R3PO training (~0.02%–0.03%).
  • How strong is the resulting constraint (§6 & Appendix B.6, Eq. 31)?: We agree—and note explicitly in §6 and Appendix B.6—that local second-order curvature α(1−α)\alpha(1-\alpha) at c=0c=0 is small when initial background mass 1−α1-\alpha is small. However, under the idealized two-pool uniform shift model Eq. 31 (App. B.6), DKL(pfcC∥pf0C)=cαc−log⁡(1−α+αec),αc=αec1−α+αec,D_{\mathrm{KL}}(p_{f_c}^C \| p_{f_0}^C) = c\alpha_c - \log(1-\alpha+\alpha e^c), \qquad \alpha_c = \frac{\alpha e^c}{1-\alpha+\alpha e^c}, using the observed mean reference mass 1−α=0.00021-\alpha = 0.0002 (τ=0.01\tau=0.01) as an illustrative parameter shows how rapidly the non-local penalty grows once hard-pool scores drift downward relative to background passages: the block-shift KL is small (0.00170.0017) for a mild shift Δs=−0.02\Delta s = -0.02 (c=Δs/τ=−2c = \Delta s/\tau = -2), rises to 0.120.12 at Δs=−0.05\Delta s = -0.05 (c=−5c=-5), and reaches 2.522.52 to 6.466.46 once a uniform downward shift reaches Δs=−0.08\Delta s = -0.08 to −0.10-0.10 (c=−8c = -8 to −10-10, comparable in scale to the 0.10–0.13 mean boundary margins reported in §7.2), where block background mass (1−αc)(1-\alpha_c) rises to 0.370.37–0.820.82 (before approaching the finite −log⁡(1−α)-\log(1-\alpha) limit noted in App. B.6). Together with Anchor-Only (47.7% EM without sampling background passages in rollouts, Table 15), this explains why a 0.0002 reference mass still provides an effective cross-pool anchor in practice.
Reviewer_2Z4BOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R3PO, a framework that trains the dense retriever in RAG with outcome-based RL. The authors use the retriever's native score directly as a softmax policy, and justify this by showing that a positive affine map is the only shared scalar map that preserves the geometry of contrastive scores (Theorem 1). The LLM reasoner is first trained with GRPO while the retriever is frozen, and then the retriever is trained with the same objective while the reasoner is frozen, using a trajectory-level reward. They also show that an objective computed only on a small hard-negative pool cannot detect a uniform score shift of that pool relative to the rest of the corpus (Theorem 2), and address this by adding randomly sampled background documents to the candidate pool. Experiments on several QA benchmarks with three retrievers show gains on average in both end-to-end EM and retriever-only Pass@3.

Critical strengths
  • Well-defined theoretical background: I found that this paper provides a well-defined theoretical background (Theorems 1 and 2), which helps explain why the framework design is needed. Specifically, Theorem 2 explains that a candidate pool of hard negatives is not adequate to catch a uniform score shift of the whole pool. I think it is impressive that the fix, mixing random background documents, comes naturally out of the same analysis. This is the main reason I see value in this paper.

  • Adaptability to other RAG systems: The framework does not require additional modules, passage-level labels, or a judge LLM, which enables adaptation to other RAG systems. Also, R3PO keeps the original scoring structure (i.e., inner product between query and document embeddings), so the trained retriever can be deployed by simply rebuilding the index. I think this makes R3PO practical for real RAG systems.

  • Empirical support for the design choices: The paper provides experiments consistent with the theoretical claims. The native score scaling performs best among the tested mappings (Table 9), and training without background documents shows no gain when moving from reranking to full-corpus retrieval (Table 11). I think this makes the theoretical claims more convincing.

Critical weaknesses
  • Limited evaluation metrics: Both the reward and the evaluation rely on EM, which is known to often fail at judging answer correctness [1, 2]. I wonder whether the authors considered a metric with better human alignment (e.g., an LLM judge) for evaluation or as the reward. Also, the retriever is evaluated only with Pass@3. I think reporting other metrics such as recall@k or nDCG@k would help, especially to support the claim that the multi-hop gains come from better retrieval of bridge documents.

  • Overlap with prior work: The paper already cites several related works (e.g., HARR, PAO, Retrieval-GRPO), but I think there are other works closely related to the pillars.

    • Policy construction: ROPG-RL [3] uses the same structure in personalized RAG. It defines the retriever policy over top candidates from the initial retriever and trains it with policy gradient using rewards from the downstream LLM.
    • Generalization: In large-scale search systems, [4] already observed that training only with top-K hard negatives fails for full-index retrieval, even though it works well for ranking a small candidate set, and proposed blending them with negatives randomly sampled from the whole document pool. This closely matches the setup and the finding in Table 11. R3PO uses random documents as anchors for the KL term rather than as negatives, but the underlying idea seems similar.

So I think each component is reasonable, but apart from the theoretical analysis, R3PO is more of a synthesis of existing works than a fundamentally new approach.

  • Limited generalizability: I'd like to see whether R3PO generalizes beyond the current setup in two aspects.
    • Query types: All datasets seem to focus on short-form answers. In practice, questions may require either short-form or long-form answers, so I'd like to see whether R3PO also works on open-ended, long-form answer datasets (e.g., ELI5 or LoTTE, following [1]). Since the reward depends on EM and answer containment, it is also unclear how the reward would be defined in such cases.
    • Corpora: I think the design for full-corpus generalization is useful. However, I wonder whether it also works on other corpora. For example, it would be helpful to check whether the performance improves or at least stays the same when the retriever is trained on corpus A (e.g., Wikipedia) and then used on an independent corpus B (e.g., any non-Wikipedia corpus such as SciFact).

Also, considering the overall cost of the two-stage RL training for the reasoner and retriever, the gains of R3PO seem marginal (about 1 to 2 points in average EM).

References

[1] Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, and Hwanjun Song. 2026. Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability. In COLM.
[2] Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023. Evaluating Open-Domain Question Answering in the Era of Large Language Models. In ACL.
[3] Alireza Salemi, Surya Kallumadi, and Hamed Zamani. 2024. Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation. In SIGIR.
[4] Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding-based Retrieval in Facebook Search. In KDD.

Questions for authors
  • I understand that better retrieval does not always lead to better generation (e.g., on TriviaQA, Pass@3 drops in Table 12 while EM increases in Table 2). However, since R3PO trains the retriever to align with the reasoner, I wonder whether it makes retrieval gains translate into generation gains more effectively. For example, it would be helpful to compare the performance when the same retrieved documents are given to the trained reasoner in R3PO and to another reasoner.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2Z4B (1/4)

Many thanks for your detailed review and valuable feedback—especially for highlighting the value of Theorems 1–2 and R3PO's practical full-corpus deployability! We believe that addressing your comments has resulted in improving the clarity and presentation of the paper's contributions and has brought the paper to a higher standard. Below we address your comments individually in order. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all equations, sections, tables, and bibliographical references refer to those in the revised paper.

1. Expanded Answer & Retrieval Metrics and Reward Modularity (W1)

In the original submission, we reported end-to-end EM (Table 2) and isolated retriever Pass@3 (Table 16, formerly Table 12, i.e., binary answer-containment Hit@3). Following your recommendation and Kamalloo et al. [2], we newly added semantic answer metrics, rank-sensitive retrieval metrics, and a multi-hop bridge audit across all benchmarks (summary in §7.1, Table 3; full per-dataset tables in App. D.1, D.4–D.5, Tables 12, 17–19):

  • Token F1 & LLM-as-a-Judge evaluation (Tables 18–19, new): Seven-dataset average Token F1 improves across all three backbones (+1.0 to +1.6 pts), and DeepSeek-V4.1-Flash Judge improves in all 14/14 dataset pairs on E5 (60.8% →\to 62.9%, with +3.0 pts / +5.4% rel. on multi-hop QA: 55.1% →\to 58.1%) and Agentic-R (61.0% →\to 62.4%), plus +0.8 pts on Qwen-0.6B (58.6% →\to 59.4%). Notably, R3PO + E5 (62.9% Judge) outperforms frozen Agentic-R (61.0% Judge) and lifts Agentic-R further to 62.4%.
  • Using human-aligned metrics as the training reward (§5 & App. E): R3PO's GRPO objective (§5) consumes the trajectory reward (Eq. 11) as a black-box scalar, so RansR_{\mathrm{ans}} can directly use an LLM-judge or claim-verifiability signal ([1]) without altering the policy optimization framework; we retained automatic string-based rewards during Stage-II training to avoid introducing online judge-LLM calls and latency overhead during rollouts (App. E).
  • Rank-sensitive nDCG@3 & multi-hop bridge evidence (Table 17 & Table 12, new): Retriever-only nDCG@3 improves in 20 of 21 pairs (+4.9%, +8.3%, and +2.9% relative on E5, Agentic-R, and Qwen-0.6B). Furthermore, because intermediate multi-hop bridges rarely contain the final answer string, we newly audited 4,317 intermediate multi-hop bridge retrieval actions (App. D.1, Table 12), showing that R3PO increases positively reinforced correct bridges by +71% (228 →\to 389) while penalizing incorrect bridges at a 3.6x negative-to-positive ratio (799 vs. 225).

Seven-dataset macro-averages (Table 3).

MethodAvg. EM (%) ↑\uparrowToken F1 (%) ↑\uparrowLLM-Judge (%) ↑\uparrowPass@3 (%) ↑\uparrownDCG@3 ↑\uparrowAvg. S ↓\downarrowLatency (s) ↓\downarrow
Search-R1 + E546.555.060.863.544.911.9351.66
R3PO + E547.8 (+2.6%)56.0 (+1.8%)62.9 (+3.5%)66.4 (+4.6%)47.09 (+4.9%)1.91 (-1.0%)50.53 (-2.2%)
Search-R1 + Agentic-R47.255.561.065.444.742.4953.07
R3PO + Agentic-R48.3 (+2.2%)56.7 (+2.1%)62.4 (+2.2%)67.6 (+3.4%)48.47 (+8.3%)2.46 (-1.2%)51.93 (-2.2%)
Search-R1 + Qwen-0.6B44.753.358.660.243.011.8454.47
R3PO + Qwen-0.6B46.2 (+3.5%)54.9 (+3.0%)59.4 (+1.3%)61.9 (+2.9%)44.26 (+2.9%)1.82 (-1.1%)53.83 (-1.2%)
AuthorsOpenReview ↗

Response to Reviewer 2Z4B (2/4)

2. Core Novelty, Uniqueness, and Distinction from ROPG-RL [3] & Facebook EBR [4] (W2)

We thank you for pointing out [3] and [4] and have integrated both into §1, §2, and Appendix A. While individual primitives have precedents ([3] defines a softmax policy over native scores; [4] mixes random and hard negatives in supervised search), R3PO is not a heuristic assembly of existing parts—it resolves a fundamental open problem in agentic RAG: how to optimize both query and document encoders of a deterministic bi-encoder via local trajectory-level RL without breaking global 21M-passage inner-product indexability.

Where R3PO's novelty and uniqueness lie:

  1. Principled duality between contrastive pretraining geometry and RL action policies (Theorem 1, Corollary 1): Rather than treating score-softmax as an ad-hoc heuristic, we prove that a positive affine map g(u)=κu+ag(u)=\kappa u+a is the unique shared pointwise policy map preserving both preference order and query-wise offset invariance (Theorem 1), with Corollary 1 fixing the pretraining-calibrated scale κ=1/τ\kappa=1/\tau from InfoNCE log-density-ratio optimality (sϕ(q,d)/(τT)s_\phi(q,d)/(\tau T) in Eq. 5)—exposing a frozen bi-encoder as a calibrated RL policy with zero auxiliary parameters and exact MIPS index compatibility (Table 13, original).
  2. Cross-pool identifiability theory for candidate-conditioned RL (Theorem 2, App. B.5–B.8): Unlike supervised learning on labeled pairs, RL updates πϕ\pi_\phi relative to a reference πϕ0\pi_{\phi_0} on small rollout pools (∣C1∣≪∣D∣|C_1|\ll|\mathcal{D}|). We formalize candidate-conditioned KL identifiability via the comparison-graph Laplacian, proving that any disconnected hard-only pool possesses an exact nullspace (fc=f0+c1Bf_c = f_0 + c\mathbf{1}_B) where uniform cross-pool drift incurs zero within-pool KL penalty yet corrupts full-corpus rankings—and showing that K3=256K_3=256 uniform anchors connect the graph so tractable 266-document RL normalization stands in for intractable 21M-corpus RL normalization.
  3. Breaking the "frozen-document" deployment barrier in multi-turn agentic RAG: Prior native-retriever RL either operated without a large external index ([3]) or froze document embeddings to avoid index invalidation (HARR). R3PO is the first framework enabling full dual-encoder (query + document) on-policy RL in multi-turn agentic search that scales to a rebuilt 21M-passage index and transfers zero-shot across agentic frameworks (R1-Searcher, Table 22, new), long-form QA (ASQA, Table 23, new), and domains (SciFact, Table 4, new).

Why [3] and [4] do not resolve this problem on their own (Tables 4, 15, 21):

  • Why ROPG-RL [3] does not transfer to 21M-passage multi-turn RAG: [3] operates in single-turn personalized generation over small, closed per-user profiles (tens to hundreds of items), where candidate scoring essentially covers the deployment set—requiring no 21M-passage index rebuild, no reference-KL regularization, and no pretraining-scale calibration. In open-domain multi-turn RAG over 21M passages, unanchored candidate-pool RL (K3=0K_3=0) fails to scale from Top-10 reranking (47.1%) to rebuilt full-corpus retrieval (47.0% vs. 47.8% with anchors in Table 15, original) and increases out-of-domain drift on SciFact (-0.014 vs. -0.003 nDCG@10 in Table 4, new).
  • Why [4]-style supervised negative mixing cannot replace R3PO's KL-anchored RL: [4] is an empirical 100:1 supervised metric-learning recipe requiring gold positive labels. In multi-turn agentic rollouts there are no step-level gold passage labels, and applying [4]-style supervised InfoNCE to successful trajectories (Trajectory Contrastive SFT (EM = 1), Table 21, new) degrades E5 from 46.5% to 45.9% EM because noisy intermediate hops are pushed up while failed rollouts (A^i<0\hat{A}_i<0) cannot push misleading hard candidates down. In R3PO's signed-advantage RL, although denominator normalization on A^i>0\hat{A}_i>0 steps shares a structural analogy with [4], failed rollouts (A^i<0\hat{A}_i<0) actively push misleading hard candidates down while unlabeled background passages anchor the cross-pool score gauge via reference KL. Crucially, our new R3PO Anchor-Only control (Table 15) never samples background passages as rollout actions—keeping them purely in the softmax denominator and KL penalty—yet achieves 47.7% full-corpus EM (vs. 47.8% standard R3PO and 47.0% for K3=0K_3=0), which supports the role of cross-pool distributional anchoring in RL.
AuthorsOpenReview ↗

Response to Reviewer 2Z4B (3/4)

3. Generalizability to Long-Form QA, Independent Corpora, and Cost vs. Gain (W3)

We added new zero-shot evaluations and cost breakdowns directly testing all three aspects of W3:

  • Open-ended / long-form QA (App. D.7, Table 23, new; App. E): Evaluated zero-shot on multi-facet long-form ASQA, R3PO + E5 improves over Search-R1 + Frozen E5 on every official metric: ROUGE-Lsum (34.58 →\to 35.14), STR-EM (40.17% →\to 41.63%), QA-F1 (30.75% →\to 32.29%), and overall DR (32.61 →\to 33.69). For training directly on open-ended datasets without short gold answers (ELI5 / LoTTE), we discuss in Appendix E replacing string-based Rans,RdocR_{\mathrm{ans}}, R_{\mathrm{doc}} with query-coverage and claim-verifiability rewards following Choi et al. [1].
  • Independent non-Wikipedia corpus (§7.2, Table 4, new): Following your exact suggestion, we evaluated our Wikipedia-trained E5 checkpoints zero-shot on biomedical SciFact. Unanchored R3PO (K3=0K_3=0) drops from frozen E5 (0.719 nDCG@10, 0.846 Recall@10) to 0.705 nDCG@10 (-0.014) and 0.826 Recall@10 (-0.020), whereas anchored R3PO (K3=256K_3=256) retains 0.716 nDCG@10 (-0.003) and 0.835 Recall@10 (-0.011), keeping independent-corpus accuracy nearly unchanged.
  • Cost vs. gain (App. C.3, Table 9 & Table 3, new): Note that Stage-I reasoner RL is already shared with the Search-R1 baseline (Table 9); R3PO adds only Stage-II retriever RL (~3 days offline on 32x H20 GPUs) plus a one-time ~4-hour re-indexing pass on 4x H20 GPUs, with no additional inference modules (-1.2% to -2.2% mean serving latency, Table 3). While strict string EM compresses the headline delta (~1.0 to 1.6 pts / +2.2%–3.5% rel.), the underlying gains in multi-hop LLM-Judge (+3.0 pts / +5.4% rel. on E5, Table 19) and nDCG@3 (+2.9% to +8.3% rel. across backbones, Table 17) are substantially larger—and match the relative gain of RL-training a 7B online reranker in CoSearch (+2.7% rel. F1) without extra serving modules.

Zero-shot long-form QA on ASQA (Table 23).

MethodROUGE-Lsum ↑\uparrowSTR-EM (%) ↑\uparrowQA-F1 (%) ↑\uparrowDR ↑\uparrow
Qwen2.5-7B-Instruct + Frozen E512.1616.7413.5312.83
Search-R1 + Frozen E534.5840.1730.7532.61
R3PO + E535.1441.6332.2933.69

Zero-shot SciFact retention and background-pool diagnostics (Table 4).

Configuration (E5)Full-Corpus QA EM (%) ↑\uparrowBg. Pool Mass ↓\downarrowSciFact nDCG@10 ↑\uparrowSciFact Recall@10 ↑\uparrow
Frozen E5 (ϕ0\phi_0)46.50.00020.7190.846
R3PO w/o Background (K3=0K_3=0)47.00.00080.7050.826
R3PO w/ Background (K3=256K_3=256)47.80.00030.7160.835
Reviewer_jYJ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes R3PO, which is a RL framework for optimizing both the reasoner and retriever in agentic RAG. The method uses dense retrieval scores as document-selection policy and applies GRPO using trajectory-level rewards. The training of R3PO consists of two stages: first optimizing the reasoner with a frozen retriever, then optimizing the retriever while keeping the reasoner fixed. The paper also provides theoretical analysis of retriever policy construction and candidate-set identifiability. Experiments show consistent improvements in exact-match accuracy and modest reductions in search calls.

Critical strengths

None

Critical weaknesses
  • Overstated novelty and incomplete comparison with baselines: Prior works such as CoSearch and MMOA-RAG have already investigated jointly optimizing reasoning and retrieval components through RL. Although CoSearch is cited, it is not experimentally compared, while MMOA-RAG is missing from the related work.
  • Limited evaluation across agentic RAG frameworks: All experiments use Search-R1 as the reasoning framework, varying only the retriever backbone. This limits the evidence supporting its broad applicability.
  • Modest gains and insufficient ablations: R3PO achieves only 1.1–1.5 absolute percentage-point improvements in average EM and approximately 1% fewer search calls, despite an additional retriever RL stage. Also, it is unclear how much of the improvement comes specifically from the proposed policy construction and background anchoring versus standard retriever RL with additional negative samples.
Questions for authors
  • How does R3PO differ from CoSearch and MMOA-RAG, which also optimize multiple RAG components through RL?
  • Can the authors clarify the individual effects of native-policy construction, trajectory-level RL, and background anchoring, and show the additional training cost?
8 comments
AuthorsOpenReview ↗

Response to Reviewer jYJ2(1/3)

Many thanks for your thoughtful review and valuable feedback! We believe that addressing your comments has resulted in improving the clarity and presentation of the paper's contributions and has brought the paper to a higher standard. Below we address your comments individually in order. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all equations, sections, tables, and bibliographical references refer to those in the revised paper.

1. Distinction & Complementarity with CoSearch and MMOA-RAG (W1, Q1)

We have cited MMOA-RAG alongside CoSearch and narrowed our framing in §1, §2, and Appendix A to clarify that R3PO is a sequentially staged framework unifying token and document policies under a shared GRPO interface (not simultaneous joint optimization). While all three apply RL in RAG, the fundamental architectural distinction is that CoSearch and MMOA-RAG train post-retrieval non-indexable generative LLM modules over a frozen embedding index, whereas R3PO directly post-trains the first-stage embedding model (bi-encoder) itself—making R3PO orthogonal and complementary to both:

  • Post-retrieval generative LLM reranking / selection over a frozen embedding model (CoSearch, MMOA-RAG): Both keep the first-stage dense bi-encoder and FAISS vector index completely frozen (CoSearch §3.1 freezes E5; MMOA-RAG §3.2, fn. 2 freezes Contriever and explicitly leaves retriever optimization to future work). Instead, they apply RL to non-indexable 7B–8B autoregressive LLM modules around the frozen index: CoSearch trains a 7B generative listwise reranker over (q,d1,…,d50)(q, d_1, \dots, d_{50}) on each search turn, while MMOA-RAG trains an 8B LLM for pre-retrieval query rewriting and post-retrieval document selection/generation over (q,d1,…,dK)(q, d_1, \dots, d_K). Because autoregressive LLM generation over concatenated (q,d1,…,dK)(q, d_1, \dots, d_K) prompts does not decompose into query-independent inner-product embeddings, it cannot pre-index documents offline: (i) for any issued query, post-retrieval recall is strictly upper-bounded by the frozen embedding model's top-KK pool, and (ii) online inference incurs extra 7B–8B LLM generation passes (per search turn in CoSearch; extra rewriter/selector calls per query in MMOA-RAG).
  • First-stage native embedding model (bi-encoder) adaptation (R3PO): R3PO directly post-trains the first-stage dense bi-encoder itself (110M–0.6B parameters, updating both query and document embeddings) and deploys a rebuilt 21M-passage FAISS index with no additional inference modules. As already shown in our original submission, expanding R3PO-adapted E5 from frozen Top-10 (46.9% EM) →\to Top-20 (47.3%) →\to Top-50 (47.7%) →\to Full-Corpus (47.8%, Table 14, formerly Table 10) yields strictly monotonic end-to-end EM gains while directly lifting isolated first-stage Pass@3 by +2.9%–4.6% relative (Table 16, formerly Table 12)—raising the first-stage retrieval ceiling that any downstream cross-encoder/reranker (such as CoSearch) depends upon. Our matched empirical baselines therefore compare native embedding-adaptation methods without extra online LLM modules (Agentic-R in Table 2, HARR and our new HARR + Our Reward in Table 5, and our new Trajectory Contrastive SFT in Table 21).

Architectural comparison with CoSearch and MMOA-RAG.

MethodModulesRL-Optimized ModulesDense Retriever Updated?First-Stage Retrieval and Downstream SelectionOnline LLM Modules Beyond Basic Retrieval + Generation
MMOA-RAGQuery Rewriter + Dense Retriever + Document Selector + GeneratorRewriter, selector, and generator (shared Llama-3-8B)NoRetrieves from the corpus for each rewritten query; selector operates on the resulting candidatesQuery rewriter and document selector
CoSearchReasoning Agent + Dense Retriever + Generative Ranker3B/7B reasoning agent and 7B rankerNoRetrieves Top-50 candidates from the corpus per search query; ranker selects/reorders these candidates7B generative ranker at each search call
R3PO (Ours)Reasoning Agent + Dense Retriever7B reasoner and 110M–0.6B shared bi-encoderYes — Stage II updates both query and document representationsAdapted bi-encoder retrieves directly from the rebuilt 21M-passage indexNo additional inference modules
AuthorsOpenReview ↗

Response to Reviewer jYJ2(2/3)

2. Applicability Beyond Search-R1 (W2)

Because R3PO preserves the standard inner-product retriever interface (Theorem 1), an R3PO-adapted dense index can be plugged into other agentic RAG frameworks without modifying or retraining them. To verify this directly, we added a new zero-shot cross-framework experiment (§7.3; Appendix D.7, Table 22): plugging our Search-R1-trained R3PO E5 index zero-shot (with zero retraining) into an unseen RL-trained agentic framework, R1-Searcher, improves 7-dataset average EM from 41.1% to 42.8% (+1.7 pts / +4.2% relative) and multi-hop QA EM by +2.2 pts (+6.0% relative: 37.6% →\to 39.8%).

Zero-shot transfer to R1-Searcher (Table 22; EM, %).

MethodNQTriviaQAPopQASingle-Hop Avg.HotpotQA2WikiMuSiQueBamboogleMulti-Hop Avg.Overall Avg.
R1-Searcher + Frozen E540.059.138.445.843.446.221.539.237.641.1
R1-Searcher + R3PO E5 (Zero-shot)40.059.341.246.8 (+2.2%)46.548.122.342.439.8 (+6.0%)42.8 (+4.2%)
AuthorsOpenReview ↗

Response to Reviewer jYJ2(3/3 Part1)

3. Magnitude of Gains, Individual Component Effects, and Additional Training Cost (W3, Q2)

Direct Answer: (a) Under semantic evaluation (DeepSeek-V4.1-Flash Judge, Table 19, new) and rank-sensitive nDCG@3 (Table 17, new), R3PO delivers +3.0 pts (+5.4% rel.) multi-hop Judge gains on E5 and +2.9%–8.3% rel. nDCG@3 gains across backbones—matching the relative F1 gain of CoSearch's 7B online reranker (+2.7% rel.) using a 110M–0.6B bi-encoder with no additional inference modules. (b) Pretraining-calibrated native scaling sϕ/(τT)s_\phi/(\tau T) outperforms uncalibrated/nonlinear policy maps (47.8% vs. 46.0%–47.1%, Table 13, original); standard contrastive negative sampling on successful rollouts (Trajectory Contrastive SFT (EM = 1), Table 21, new) degrades EM from 46.5% to 45.9%; and Anchor-Only (Table 15, new, 47.7% EM) supports the role of cross-pool anchoring without ever sampling background passages as actions. (c) Stage-II is a one-time offline cost requiring no additional inference modules at deployment, with observed 7-dataset mean multi-turn latency decreasing by 1.2%–2.2% (Table 3, new).

  • (a) Putting the +1.0–1.6 EM-point (+2.2%–3.5% rel.; 1.1–1.5 pts on rounded averages) gain in context:

    1. Already in the original submission (Table 2, Table 16): Starting from a strong veRL 0.7.0 Search-R1 baseline (App. C.1), R3PO + general-purpose E5 (47.8% EM) already outperforms frozen Agentic-R (47.2% EM)—a specialized retriever trained with complex LLM-annotated supervision—and further boosts Agentic-R to 48.3% EM, with zero regressions across all 21/21 dataset-backbone pairs (~51.7k test questions) and up to +4.6% relative Pass@3 (+9.4 pts on PopQA).
    2. Newly added metrics (§7.1, Table 3; App. D.4–D.5, Tables 17–19) & CoSearch reference point: Strict exact-string EM understates both retrieval-ranking and semantic answer gains. On DeepSeek-V4.1-Flash LLM-Judge, R3PO wins 14/14 dataset pairs on E5 and Agentic-R, gaining +2.1 pts (+3.5% rel.) overall and +3.0 pts (+5.4% rel.) on E5 multi-hop QA (Table 19), while rank-sensitive nDCG@3 rises by +4.9%, +8.3%, and +2.9% relative across the three backbones (Table 17). Notably, in CoSearch (Zeng et al., 2026), RL-training a 7B generative reranker over Top-50 yields +2.7% relative Token F1 over a fixed ranker—the same relative magnitude as R3PO's +1.8%–3.0% relative Token F1 (Table 3), achieved here by a 110M–0.6B bi-encoder with no additional inference modules.
  • (b) Individual effects of the three components (vs. standard RL / negative sampling): Several component ablations were already in Appendix D of our original submission, and we added three new controls to rule out simpler negative-sampling or SFT explanations:

    1. Native-policy construction (Table 13, formerly Table 9, original): Pretraining-calibrated linear scaling sϕ/(τT)s_\phi/(\tau T) (47.8% EM) outperforms uncalibrated linear scales sϕ/(5τT)s_\phi/(5\tau T) (46.0%) and 2sϕ/(τT)2s_\phi/(\tau T) (46.8%) as well as nonlinear sϕ5/(τT)s_\phi^5/(\tau T) (47.1%).
    2. Trajectory-level RL (Table 10, formerly Table 8, original; Table 21 & Table 5, new): Originally, Table 10 showed RansR_{\mathrm{ans}} alone lifts E5 from 46.5% →\to 47.1% and full trajectory reward reaches 47.8%. Newly added Trajectory Contrastive SFT (EM = 1) (Table 21) trains E5 with standard InfoNCE treating passages from EM = 1 rollouts (Rans=1R_{\mathrm{ans}}=1) as positives against non-selected negatives under the same rollout budget; this degrades EM from 46.5% to 45.9% due to noisy intermediate-hop passages, whereas R3PO's signed GRPO advantages achieve 47.8% (+1.8 pts over SFT). On Qwen-0.6B (Table 5), R3PO (46.2%) also outperforms query-encoder-only RL across all settings: original HARR (Query + history, 44.2%), state-matched HARR (Query only, 45.5%), and our new state- and reward-matched HARR + Our Reward (Query only, 45.6%).
    3. Background anchoring (Table 15, formerly Table 11, original + new Anchor-Only row): Originally, Table 15 showed that without background documents (K3=0K_3=0), full-corpus retrieval (47.0%) fails to improve over Top-10 reranking (47.1%), whereas K3=256K_3=256 scales from 46.9% →\to 47.8%. Crucially, R3PO does not use the 256 random passages as supervised negative labels or rely on sampling them as rollout actions: our new Anchor-Only ablation (Table 15) never samples background passages during rollouts (keeping them solely in the softmax denominator and KL penalty) and still achieves 47.7% EM, which supports the role of cross-pool anchoring.
Reviewer_9g42OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

R3PO first trains a reasoner with a frozen retriever, then freezes the reasoner and trains the retriever with GRPO. At each search step, the retriever samples a document from the original top-10 results plus 256 random background documents. Training uses trajectory rewards based on answer correctness, answer containment, and search count. The background documents aim to constrain score shifts that training within a small candidate pool cannot detect. The adapted retriever is deployed through a rebuilt full-corpus index.

Critical strengths
  1. The paper explains a useful problem: better ranking within a small candidate pool does not necessarily lead to better full-corpus retrieval. The analysis and background-document ablation provide a clear motivation for addressing this gap.

  2. Average EM improves across three retrievers and seven QA datasets, and retrieval recall also improves. The method preserves the standard dense-retrieval interface after rebuilding the index.

Critical weaknesses
  1. The training signal does not distinguish the contribution of each retrieval step. All retrieval actions in a trajectory share one advantage. A useful retrieval can receive negative feedback if later reasoning fails, while an unhelpful retrieval can receive positive feedback if the agent later recovers. The answer-containment reward also misses useful bridge documents that do not contain the final answer. Appendix E acknowledges this limitation, but the experiments do not show how much it affects learning. This matters for the claimed benefit on multi-hop tasks.

  2. The key design choices need stronger experimental support. Table 11 supports adding background documents, but also expands the candidate pool from 10 to 266. It does not isolate score anchoring from broader exploration, and no score-drift measurements are reported.

Questions for authors
  1. Can a small trajectory analysis show how often useful bridge retrievals receive negative advantages, or unhelpful retrievals receive positive ones? How much does the answer-containment reward help with this problem?

  2. Does the benefit of background documents remain when candidate-pool size is held fixed? Can you measure whether they actually reduce score drift relative to documents outside the hard pool?

  3. Which reward, rollout budget, and trainable parameters were used for query-only HARR? Does R3PO still improve under matched settings, and are the gains stable across seeds?

  4. What are the training cost, index-rebuilding cost, and end-to-end latency? About 1% fewer search calls alone does not establish a meaningful speed benefit.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9g42 (1/4)

Many thanks for your constructive review and valuable feedback! We believe that addressing your comments has resulted in improving the clarity and presentation of the paper's contributions and has brought the paper to a higher standard. Below we address your comments individually in order. The corresponding modifications in the paper are highlighted in blue.

1. Trajectory-Level Credit Assignment & Multi-Hop Bridge Audits (W1, Q1)

Direct Answer: Across 8,072 retrieval actions (Table 11, new) and 4,317 intermediate multi-hop bridge steps (Table 12, new) in Appendix D.1, our composite reward (+Rdoc−Nsearch+R_{\mathrm{doc}}-N_{\mathrm{search}}) nearly doubles active non-zero advantage coverage (26.4% →\to 46.6%), increases positively reinforced useful retrievals by +73% (1,039 →\to 1,799), and suppresses incorrect multi-hop bridges at a 3.6x negative-to-positive ratio (799 penalized vs. 225 positively reinforced).

Specifically, to quantify the step-level credit sign rates noted qualitatively in original Appendix E (alongside the end-to-end reward ablation in original Table 10, formerly Table 8: 46.5% →\to 47.1% →\to 47.2% →\to 47.8% EM), we audited G=5G=5 Stage-II rollouts:

  • Overall step utility (Table 11, new; 8,072 retrieval actions: 5,355 useful vs. 2,717 unhelpful labeled by DeepSeek-V4.1-Flash):

    • Under RansR_{\mathrm{ans}} (EM) alone, 73.6% of actions receive zero advantage (A^i=0\hat{A}_i=0); useful retrievals get A^i<0\hat{A}_i<0 in 9.7% of cases, while unhelpful retrievals get A^i>0\hat{A}_i>0 in 3.9% of cases.
    • Adding answer containment (+λdocRdoc+\lambda_{\mathrm{doc}}R_{\mathrm{doc}}) and search penalty (−λsearchNsearch-\lambda_{\mathrm{search}}N_{\mathrm{search}}) breaks outcome ties, expanding active advantages (A^i≠0\hat{A}_i\neq 0) from 26.4% →\to 39.1% →\to 46.6%, lifting A^i>0\hat{A}_i>0 on useful retrievals from 19.4% →\to 28.1% →\to 33.6% and A^i<0\hat{A}_i<0 on unhelpful retrievals from 17.4% →\to 27.6% →\to 31.4%, while monotonically raising the Pearson correlation between continuous raw A^i\hat{A}_i and step utility from 0.213 →\to 0.236 →\to 0.247.
    Reward VariantUseful A^i>0\hat{A}_i>0Useful A^i=0\hat{A}_i=0Useful A^i<0\hat{A}_i<0Unhelpful A^i>0\hat{A}_i>0Unhelpful A^i=0\hat{A}_i=0Unhelpful A^i<0\hat{A}_i<0Active (A^i≠0\hat{A}_i\neq 0)Pearson r↑r\uparrow
    RansR_{\mathrm{ans}} (EM only)19.4%70.9%9.7%3.9%78.8%17.4%26.4%0.213
    Rans+λdocRdocR_{\mathrm{ans}}+\lambda_{\mathrm{doc}}R_{\mathrm{doc}}28.1%59.8%12.1%9.1%63.3%27.6%39.1%0.236
    Full (Rans+λdocRdoc−λsearchNsearchR_{\mathrm{ans}}+\lambda_{\mathrm{doc}}R_{\mathrm{doc}}-\lambda_{\mathrm{search}}N_{\mathrm{search}})33.6%50.5%15.9%9.5%59.2%31.4%46.6%0.247
  • Intermediate multi-hop bridges (Table 12, new):

    • You rightly noted that useful Hop-1 bridge passages rarely contain the final answer string. Under RansR_{\mathrm{ans}} alone, 77.4% of bridge steps get A^i=0\hat{A}_i=0, leaving correct bridges nearly tied between A^i>0\hat{A}_i>0 (228, 13.0%) and A^i<0\hat{A}_i<0 (222, 12.7%).
    • Why trajectory-level RdocR_{\mathrm{doc}} and −Nsearch-N_{\mathrm{search}} still drive multi-hop gains: Because RdocR_{\mathrm{doc}} is evaluated over the entire trajectory, retrieving a valid Hop-1 bridge enables Hop-2 to surface an answer-bearing passage (Rdoc=1R_{\mathrm{doc}}=1) in fewer searches. Consequently, adding +Rdoc+R_{\mathrm{doc}} and −Nsearch-N_{\mathrm{search}} boosts positively reinforced correct bridges by +71% (228 →\to 346 →\to 389). While shared trajectory credit also assigns A^i<0\hat{A}_i<0 to 408 (23.3%) correct bridges when a rollout finds a valid Hop-1 bridge yet fails on Hop-2, candidate-set softmax updates still favor valid bridges in expectation: whereas correct bridges have near-neutral sign balance (22.2% positive vs. 23.3% negative, 389 vs. 408), incorrect bridges face a heavy negative skew (31.1% penalized vs. 8.8% positive—a 3.6x negative-to-positive ratio, 799 penalized vs. 225 false-positive, up from 433 under RansR_{\mathrm{ans}}), strongly pushing down competing incorrect bridges. This aligns with our multi-hop EM gains (+3.1%–5.4% rel., Table 2) and multi-hop nDCG@3 gains (+3.9%–7.9% rel., Table 17), while we transparently report the remaining step-credit noise in Appendix D.1 and Appendix E.
    Reward VariantCorrect Bridge A^i>0\hat{A}_i>0Correct A^i=0\hat{A}_i=0Correct A^i<0\hat{A}_i<0Incorrect Bridge A^i>0\hat{A}_i>0Incorrect A^i=0\hat{A}_i=0Incorrect A^i<0\hat{A}_i<0Active (A^i≠0\hat{A}_i\neq 0)
    RansR_{\mathrm{ans}}(EM only)13.0%74.3%12.7%3.6%79.5%16.9%22.6%
    Rans+λdocRdocR_{\mathrm{ans}}+\lambda_{\mathrm{doc}}R_{\mathrm{doc}}19.8%58.5%21.8%8.0%64.2%27.8%38.1%
    Rans+λdocRdoc−λsearchNsearchR_{\mathrm{ans}}+\lambda_{\mathrm{doc}}R_{\mathrm{doc}}-\lambda_{\mathrm{search}}N_{\mathrm{search}})22.2%54.5%23.3%8.8%60.1%31.1%42.2%
AuthorsOpenReview ↗

Response to Reviewer 9g42 (2/4)

2. Isolating Score Anchoring at Fixed Pool Size & Measuring Score Drift (W2, Q2)

Direct Answer: Yes to both. (1) When the rollout action-sampling support is held strictly fixed at 10 documents (identical to K3=0K_3=0, never sampling background passages as actions), keeping the 256 background documents solely in the softmax denominator and reference KL penalty (Anchor-Only) retains 47.7% full-corpus EM (vs. 47.8% standard R3PO and 47.0% unanchored K3=0K_3=0, Table 15), which supports the role of cross-pool anchoring. (2) Direct diagnostics (§7.2, Table 4) show that background anchoring cuts outside-pool probability-mass drift from frozen E5 (0.0002) by 6x (+0.0001 to 0.0003 vs. +0.0006 to 0.0008—a 4x jump without anchors), maintains a +30% wider native cosine boundary margin (0.13 vs. 0.10), and better preserves zero-shot cross-corpus retrieval on SciFact (nDCG@10 drop of 0.003 vs. 0.014; Recall@10 drop of 0.011 vs. 0.020).

In detail, building on original Table 15 (formerly Table 11, where K3=0K_3=0 failed to scale from 47.1% Top-10 reranking to 47.0% full-corpus retrieval whereas K3=256K_3=256 scaled from 46.9% →\to 47.8%), we added two new sets of experiments (§7.2, Table 4 and App. D.3, Table 15):

  • Holding rollout action-sampling support fixed at 10 documents (Anchor-Only, Table 15, new): During Stage-II rollouts, R3PO Anchor-Only samples each document action (K2=1K_2=1) strictly from Top10(q)\mathrm{Top}_{10}(q) (holding the action-sampling support fixed at 10 documents, identical to K3=0K_3=0), while keeping the 256 random corpus passages only inside the softmax denominator and reference KL penalty. Anchor-Only achieves 47.7% full-corpus EM—retaining nearly the entire gain of standard R3PO (47.8%) over K3=0K_3=0 (47.0%) and matching or exceeding standard R3PO on all four multi-hop benchmarks. (Consistent with this, on the evaluated checkpoints and diagnostic pools in Table 4, the mean total selection probability of all 256 background documents is only about 0.02%–0.03%, i.e., 0.0002–0.0003.) Retaining 47.7% EM without passing background documents to the reasoner supports the role of cross-pool anchoring over rollout exploration.
  • Direct measurement of cross-pool separation & drift (§7.2, Table 4, new): Evaluating checkpoints on fixed diagnostic mixed pools (Top10(q)∪Bg256\mathrm{Top}_{10}(q)\cup\mathrm{Bg}_{256}):
    1. Native cosine boundary margin: E[min⁡d∈Top10sϕ(q,d)−max⁡u∈Bg256sϕ(q,u)]\mathbb{E}[\min_{d\in\mathrm{Top}_{10}}s_\phi(q,d)-\max_{u\in\mathrm{Bg}_{256}}s_\phi(q,u)] is 0.13 with background training (K3=256K_3=256) versus 0.10 without it (K3=0K_3=0, +30%), showing stronger separation from outside passages.
    2. Background probability-mass drift: Unanchored training (K3=0K_3=0) increases mean total background probability mass 4x from 0.0002 (frozen E5) to 0.0008 (+0.0006 drift), whereas anchored R3PO constrains it to 0.0003 (+0.0001 drift, 6x smaller).
    3. Out-of-corpus retention: On zero-shot biomedical SciFact (Table 4), anchored R3PO keeps retrieval much closer to frozen E5 (nDCG@10: 0.716 vs. 0.705 for K3=0K_3=0, relative to 0.719 frozen; Recall@10: 0.835 vs. 0.826, relative to 0.846 frozen).

Background ablation and Anchor-Only control (Table 15; EM, %).

TrainingNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Top-10 reranking: K3=0K_3=049.765.748.449.647.923.544.847.1
Top-10 reranking: K3=256K_3=25649.566.346.948.545.922.948.046.9
Full-corpus retrieval: K3=0K_3=049.665.948.049.147.022.447.247.0
Full-corpus retrieval: Anchor-Only (no bg. sampling)49.465.748.749.548.123.648.847.7
Full-corpus retrieval: K3=256K_3=25650.165.949.149.347.523.648.847.8

Background-pool probability mass and SciFact retention (Table 4).

Configuration (E5)Full-Corpus QA EM (%) ↑\uparrowBg. Pool Mass ↓\downarrowSciFact nDCG@10 ↑\uparrowSciFact Recall@10 ↑\uparrow
Frozen E5 (ϕ0\phi_0)46.50.00020.7190.846
R3PO w/o Background (K3=0K_3=0)47.00.00080.7050.826
R3PO w/ Background (K3=256K_3=256)47.80.00030.7160.835
AuthorsOpenReview ↗

Response to Reviewer 9g42 (3/4)

3. Matched HARR Configuration & Seed Stability (Q3)

In the original submission (Table 5, formerly Table 3), we compared R3PO (46.2% EM) against original HARR (Query + history) (44.2%) and state-matched HARR (Query only) (45.5%) on Qwen-0.6B, leaving three details unclear that we now clarify and expand in §7.3, Table 5 and Appendix C.3 (new):

  • Matched rollout budget & trainable parameters: All Qwen-0.6B runs share the exact same Stage-I reasoner, merged NQ+HotpotQA training set, batch size (512), rollout budget (G=5G=5), and 1 training epoch. Following HARR's original design (which freezes document embeddings to avoid corpus re-indexing), HARR performs full-parameter training of the query encoder with a frozen document encoder/index, whereas R3PO updates the shared bi-encoder for both queries and documents and rebuilds the full-corpus index.
  • New reward-matched baseline (HARR + Our Reward, Table 5): Replacing HARR's original reward with R3PO's exact composite reward (Eq. 11) reaches 45.6% EM, and R3PO still improves to 46.2% (+0.6 pts over state- and reward-matched HARR; +2.0 pts over original HARR), isolating the gain from dual-encoder adaptation and anchored full-corpus re-indexing.
  • Evaluation-seed & training-level stability (Table 5, new): Across two independent multi-turn evaluation seeds across all seven benchmarks, R3PO achieves 46.21% and 46.20% EM (given the ~2,304 GPU-h cost per Stage-II run in Table 9, these two runs measure multi-turn inference stability across evaluation seeds rather than separate Stage-II training runs, while training-level stability is corroborated by consistent rebuilt-index gains across all three backbones in Table 2 and the independent Anchor-Only training run in Table 15).

Matched HARR controls and evaluation seeds (Table 5; EM, %).

MethodRL StateNQTriviaQAPopQASingle-Hop Avg.HotpotQA2WikiMuSiQueBamboogleMulti-Hop Avg.Overall Avg.
FrozenQuery + history46.764.948.153.244.945.016.438.436.243.5
HARRQuery + history46.565.048.553.3 (+0.2%)46.547.018.337.637.4 (+3.2%)44.2 (+1.6%)
FrozenQuery only46.464.147.352.646.647.419.241.638.744.7
HARRQuery only47.064.947.353.1 (+0.9%)47.047.119.845.639.9 (+3.0%)45.5 (+2.0%)
HARR + Our RewardQuery only47.364.847.753.3 (+1.3%)47.147.220.544.839.9 (+3.1%)45.6 (+2.2%)
R3PO (Eval Seed 1)Query only47.564.548.353.4 (+1.6%)47.747.720.647.240.8 (+5.4%)46.2 (+3.5%)
R3PO (Eval Seed 2)Query only47.964.848.153.6 (+1.9%)47.847.620.846.440.7 (+5.0%)46.2 (+3.5%)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.