- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
R3PO uses a two-stage GRPO framework to train the reasoner and dense retriever in agentic RAG. It turns the retriever’s native inner-product scores into a document-selection policy and updates that policy using rewards from completed QA trajectories. The paper shows that training on a small candidate pool cannot detect a uniform score shift relative to documents outside the pool, and adds randomly sampled background documents as anchors. Across seven QA datasets and three retriever backbones, the authors report improvements in answer exact match (EM) and retriever Pass@3. An E5 ablation compares candidate-pool reranking with retrieval from a rebuilt full-corpus index.
Critical strengths
- The paper identifies a concrete deployment problem. Its candidate-set identifiability analysis explains a specific failure mode and motivates cross-pool comparisons. This is the paper’s most compelling contribution.
- The evaluation includes full-corpus retrieval. The authors re-encode the corpus and rebuild the index rather than evaluating only within a fixed candidate pool. Reporting both end-to-end EM and retriever Pass@3 provides initial evidence that the method improves retrieval itself.
Critical weaknesses
- The evidence for the central mechanism is incomplete. The key background-document ablation is reported only for E5. The paper does not directly measure the proposed cross-pool score drift or report variation across random seeds. Table 11 is consistent with the theory, but does not establish that anchoring works through the proposed mechanism or justify the claim that it ensures full-corpus generalization. This limits confidence in the paper’s main contribution.
- The novelty and “unified end-to-end” claims need narrower framing. Prior work has already applied stochastic document selection and GRPO to dense retrievers [1]. Moreover, R3PO trains its two components sequentially with different rewards, rather than jointly optimizing them. Its distinctive contribution appears to be the cross-pool analysis and anchoring strategy; the current framing makes that contribution harder to assess precisely.
[1] Y. Zhang et al., “Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG,” 2026.
Questions for authors
- In the two conditions in Table 11, how do the scores, ranks, or probability mass of the Top-10 documents change relative to documents outside that pool after training? Is there direct evidence of the uniform cross-pool drift described by Theorem 2?
- How much probability mass do background documents typically receive in the training softmax? If that mass is very small, how strong is the resulting constraint on cross-pool drift?
4 comments
Response to Reviewer RXrw (2/4)
2. Narrower Framing of "Unified", Sequential Training, and Prior Retriever RL (W2)
We agree completely and revised Abstract, §1, §2, §3, §5, §6, Appendix A, and Appendix E to:
- Explicitly state upfront that R3PO is a sequentially staged two-stage framework with stage-specific trajectory rewards (not simultaneous joint optimization), and that "unified" refers strictly to casting both token generation and native document selection under a shared rollout–outcome–GRPO policy interface;
- Credit HARR
[1](alongsideROPG-RLandPAO) prominently in§1–§2andAppendix Afor stochastic policy-gradient/GRPO retriever training, so R3PO's distinctive contribution in agentic RAG—adapting both query and document representations via the shared bi-encoder and anchoring cross-pool drift for rebuilt full-corpus deployment rather than freezing the document index—can be assessed cleanly; and - Replace "ensures generalization" with "constrains cross-pool score drift" / "generalizes reliably".
Response to Reviewer RXrw (3/4)
3. Score Separation & Probability-Mass Shifts Across the Two Conditions of Original Table 11 (Q1)
Direct Answer: Yes—direct diagnostics show systematic cross-pool drift between and outside documents when anchors are omitted. Evaluating Frozen E5 (), Unanchored R3PO (), and Anchored R3PO () on identical fixed diagnostic mixed pools (§7.2 and Table 4, new) shows that without anchors (), outside background probability mass jumps 4x from 0.0002 to 0.0008 as the Top-10 boundary margin narrows to 0.10, whereas anchoring () constrains background mass to 0.0003 (6x smaller drift, +0.0001 vs. +0.0006) and maintains a +30% wider boundary margin (0.13):
- Native cosine boundary margin (
§7.2):Theorem 2proves that a uniform block shift between and outside documents lies in the exact nullspace of the objective (App. B.5); in practice, parameterized shared-encoder SGD manifests this unpenalized direction as systematic cross-pool encroachment. Directly measuring shows that anchored training () maintains a+30%wider mean boundary margin of 0.13 versus 0.10 for unanchored training (). - Relative probability-mass shift (
Table 4): Under frozen E5 (), the 256 background passages receive a mean total softmax mass of 0.0002 (so holds0.9998). Training without background anchors () causes outside-pool mass to drift upward4xto 0.0008 (+0.0006drift), whereas background-trained R3PO () constrains it to 0.0003 (+0.0001drift,6xsmaller)—and better preserves zero-shot retrieval on out-of-corpus biomedicalSciFact(nDCG@10: 0.716 vs. 0.705, compared with0.719frozen;Recall@10: 0.835 vs. 0.826, compared with0.846frozen).
Cross-pool probability diagnostics and SciFact retention (Table 4).
| Configuration (E5) | Full-Corpus QA EM (%) | Bg. Pool Mass | SciFact nDCG@10 | SciFact Recall@10 |
|---|---|---|---|---|
| Frozen E5 () | 46.5 | 0.0002 | 0.719 | 0.846 |
| R3PO w/o Background () | 47.0 | 0.0008 | 0.705 | 0.826 |
| R3PO w/ Background () | 47.8 | 0.0003 | 0.716 | 0.835 |
Response to Reviewer RXrw (4/4)
4. Softmax Probability Mass of Background Documents & Strength of the Constraint (Q2)
- Measured softmax probability mass (
Table 4, new): On the evaluated checkpoints and diagnostic mixed pools (), all 256 background documents combined receive a mean total selection probability of 0.0002 under frozen E5 and 0.0003 after anchored R3PO training (~0.02%–0.03%). - How strong is the resulting constraint (
§6&Appendix B.6, Eq. 31)?: We agree—and note explicitly in§6andAppendix B.6—that local second-order curvature at is small when initial background mass is small. However, under the idealized two-pool uniform shift modelEq. 31(App. B.6), using the observed mean reference mass () as an illustrative parameter shows how rapidly the non-local penalty grows once hard-pool scores drift downward relative to background passages: the block-shift KL is small () for a mild shift (), rises to at (), and reaches to once a uniform downward shift reaches to ( to , comparable in scale to the0.10–0.13mean boundary margins reported in§7.2), where block background mass rises to – (before approaching the finite limit noted inApp. B.6). Together withAnchor-Only(47.7%EM without sampling background passages in rollouts,Table 15), this explains why a0.0002reference mass still provides an effective cross-pool anchor in practice.
Response to Reviewer RXrw (1/4)
Many thanks for your positive review and valuable feedback! We believe that addressing your comments has resulted in improving the clarity and presentation of the paper's contributions and has brought the paper to a higher standard. Below we address your comments individually in order. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all equations, sections, tables, and bibliographical references refer to those in the revised paper.
1. Strengthening Evidence for the Central Anchoring Mechanism & Stability (
W1)In the original submission, we reported rebuilt full-corpus retrieval gains across all three backbones (
E5,Agentic-R,Qwen-0.6B) on end-to-end EM (Table 2) and isolatedPass@3(Table 16, formerlyTable 12), together with the E5 K3=0 vs. K3=256 reranking-to-retrieval reversal (Table 15, formerlyTable 11). You rightly noted that originalTable 11alone did not isolate the anchoring mechanism from rollout sampling, lacked direct cross-pool drift measurements and seed-stability checks, and used overly strong "ensures generalization" wording. In the revised paper, we added:Anchor-Onlyablation isolating loss anchoring (App. D.3, Table 15): During Stage-II rollouts,Anchor-Onlysamples actions strictly from Top10(q) (never passing background passages to the reasoner), keeping the 256 random corpus passages only in the softmax denominator and reference KL penalty. Its 47.7% full-corpus EM (vs.47.8%standard R3PO and47.0%for K3=0, matching or exceeding standard R3PO on all four multi-hop datasets) supports the role of cross-pool anchoring directly through the training loss.SciFactdiagnostics (§7.2, Table 4): Detailed underQ1below.§7.1–7.3, Tables 2, 3, 5;App. C.3): Rebuilt full-corpus gains hold consistently across all three backbones (E5,Agentic-R,Qwen-0.6B) on originalEM(Table 2) andPass@3(Table 16) as well as newToken F1,LLM-Judge, andnDCG@3(Table 3). Crucially,Table 5(expanded in blue) provides a direct controlled validation on a second backbone (Qwen-0.6B): onceHARRis strictly aligned with R3PO'sQuery onlystate (45.5%, original) and exact composite reward (HARR + Our Reward,45.6%, new) under the same G=5 budget, the remaining difference is precisely freezing the document index versus R3PO's background-anchored dual-encoder adaptation and rebuilt full-corpus index—lifting accuracy further to 46.2% EM (46.21%and46.20%across two independent evaluation seeds; given the ~2,304 GPU-h cost per Stage-II run, these two runs measure multi-turn inference stability across evaluation seeds, while training-level stability is corroborated by consistent gains across all three backbones and the independentAnchor-Onlytraining run inTable 15).W2, we replaced unconditional "ensures generalization" phrasing with empirically scoped statements aligned withTheorem 2andAppendix B.6–B.8.