- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper identifies an "information-credit gap" in agentic RL for retrieval-augmented QA, where failures caused by poor evidence are wrongly attributed to the LLM. The authors show empirically that retrieval and policy learning are order-sensitive—adapting the retriever first yields larger gains—and formulate joint training as a bilevel optimization problem. They propose BRIDGE, a first-order single-state penalty method motivated by a loss-landscape analysis, implemented with a query-side LoRA adapter.
Critical strengths
- The empirical study comparing RAG-then-RL and RL-then-RAG provides a clear, well-motivated justification for treating retrieval adaptation as the lower-level problem.
- The evaluation covers seven open-domain and five medical question-answering benchmarks, multiple backbone sizes, in-domain and out-of-domain settings, and several retrieval and reinforcement-learning baselines.
Critical weaknesses
-
The main-text training workflow is hard to follow. The exposition does not make it clear, in a single place, what actually happens after a rollout is sampled: in what order the RL reward, retrieval-turn credit, RAG likelihood, retriever LoRA update (every P rounds), and policy update are computed, and which quantities are reused across the two updates. Algorithm 1 is helpful but underspecified relative to the surrounding prose, and the reader has to reconstruct the pipeline by jumping between Section 3.2, Eqs. (11)–(12), and Appendix A.1.
-
The necessity of the bilevel joint training in Eq. (4) is not justified against simpler alternatives. It is unclear why and must be optimized jointly within a bilevel formulation rather than in a decoupled fashion — e.g., freezing one while optimizing the other, or alternating in epochs. Two specific concerns follow:
- When is used to train via , how is the gradient actually backpropagated, and does the gold answer appear in the LLM's conditioning context in any form during training? If so, this risks "answer leakage," where the model learns to exploit the ground-truth signal rather than to reason. The authors should clarify the information flow and rule this out explicitly.
- Equation (12) states that participates in the optimization of , yet Table 3 does not isolate the effect of this term on : there is no variant that removes the RAG contribution to the policy update while keeping it for the retriever (or vice versa). An ablation that cleanly separates "RAG → only" vs. "RAG → only" vs. "RAG → both" is needed to support the design.
- Overall, the authors should (i) justify the bilevel setting against a decoupled baseline with matched compute, and (ii) extend the ablation to show the specific contribution of to the update.
-
The derivation in Eqs. (8)–(10) appears to add complexity without adding content. Starting from the penalty , a direct rearrangement gives , and taking on both sides already yields what appears in Eq. (10). The two-step Taylor-expansion / optimality-condition argument between (9) and (10) therefore does not seem to deliver a result different from this one-line manipulation. If the intermediate approximations are in fact doing non-trivial work (e.g., bounding error terms, enabling the rate, or legitimizing the subsequent "drop both RAG terms" step), that should be stated explicitly; otherwise the derivation reads as padded, and the suspicion that
it was generated or inflated by an AIis hard to dispel. -
The "adaptive to the retriever" claim rests on a single retriever. All experiments use E5-base-v2 as the base retriever, which makes it impossible to tell whether BRIDGE's gains generalize or whether the method is implicitly co-designed with E5's embedding space. The paper's central thesis —
the retriever should first be optimized to make useful evidence available, and the LLM should then learn to search and reason with that evidence (line 49)— is a claim about retrievers in general, so it should be supported with a broader set of base retrievers (e.g., BGE, Contriever, GTE, or a sparse baseline such as BM25 combined with a dense reranker). At minimum, one additional dense retriever and one qualitatively different retriever (sparse or hybrid) should be reported on the open-domain QA suite, so that the order-sensitivity finding and the BRIDGE gains can be shown to transfer.
Questions for authors
see weakness