acceptodds
Under review as a conference paper at ICLR 2027

BRIDGE: Bilevel Retrieval Credit-Aware Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) with verifiable reward enhances the capability of large language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and reasoning. However, most existing ARL methods optimize only LLM generated tokens and treat external retrieved evidence as environment observations, creating an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than the retriever and motivating the joint training over LLM and retrieval model. In this paper, we show that retrieval and LLM policy learning are order-sensitive, with retriever adaptation followed by policy optimization yielding the largest reward gain. To preserve this hierarchy while allowing both components to co-adapt, we formulate retrieval-augmented agentic RL as a bilevel optimization problem. To solve this bilevel problem efficiently, we introduce Bilevel Retrieval via Information-Directed Gradient Estimation (BRIDGE), a memory-efficient first-order method motivated by the loss landscape analysis of the RL and retrieval objectives. Across seven open-domain QA benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving multi-hop accuracy by 9.8% and 3.3%, respectively. It also achieves the best macro-averaged answer accuracy and reasoning quality across five medical QA benchmarks.

Primary area:
optimization
Keywords:
bilevel optimization, agentic reinforcement learning, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_HN7GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies an "information-credit gap" in agentic RL for retrieval-augmented QA, where failures caused by poor evidence are wrongly attributed to the LLM. The authors show empirically that retrieval and policy learning are order-sensitive—adapting the retriever first yields larger gains—and formulate joint training as a bilevel optimization problem. They propose BRIDGE, a first-order single-state penalty method motivated by a loss-landscape analysis, implemented with a query-side LoRA adapter.

Critical strengths
  1. The empirical study comparing RAG-then-RL and RL-then-RAG provides a clear, well-motivated justification for treating retrieval adaptation as the lower-level problem.
  2. The evaluation covers seven open-domain and five medical question-answering benchmarks, multiple backbone sizes, in-domain and out-of-domain settings, and several retrieval and reinforcement-learning baselines.
Critical weaknesses
  1. The main-text training workflow is hard to follow. The exposition does not make it clear, in a single place, what actually happens after a rollout is sampled: in what order the RL reward, retrieval-turn credit, RAG likelihood, retriever LoRA update (every P rounds), and policy update are computed, and which quantities are reused across the two updates. Algorithm 1 is helpful but underspecified relative to the surrounding prose, and the reader has to reconstruct the pipeline by jumping between Section 3.2, Eqs. (11)–(12), and Appendix A.1.

  2. The necessity of the bilevel joint training in Eq. (4) is not justified against simpler alternatives. It is unclear why θ\theta and η\eta must be optimized jointly within a bilevel formulation rather than in a decoupled fashion — e.g., freezing one while optimizing the other, or alternating in epochs. Two specific concerns follow:

  • When LRAGL_{\text{RAG}} is used to train θ\theta via pθ(y∣h,d)p_\theta(y \mid h, d), how is the gradient actually backpropagated, and does the gold answer yy appear in the LLM's conditioning context in any form during training? If so, this risks "answer leakage," where the model learns to exploit the ground-truth signal rather than to reason. The authors should clarify the information flow and rule this out explicitly.
  • Equation (12) states that LRAGL_{\text{RAG}} participates in the optimization of θ\theta, yet Table 3 does not isolate the effect of this term on θ\theta: there is no variant that removes the RAG contribution to the policy update while keeping it for the retriever (or vice versa). An ablation that cleanly separates "RAG → η\eta only" vs. "RAG → θ\theta only" vs. "RAG → both" is needed to support the design.
  • Overall, the authors should (i) justify the bilevel setting against a decoupled baseline with matched compute, and (ii) extend the ablation to show the specific contribution of LRAGL_{\text{RAG}} to the θ\theta update.
  1. The derivation in Eqs. (8)–(10) appears to add complexity without adding content. Starting from the penalty Lγ=LRL+γLRAGL_\gamma = L_{\text{RL}} + \gamma L_{\text{RAG}}, a direct rearrangement gives LRL=Lγ−γLRAGL_{\text{RL}} = L_\gamma - \gamma L_{\text{RAG}}, and taking ∇θ\nabla_\theta on both sides already yields what appears in Eq. (10). The two-step Taylor-expansion / optimality-condition argument between (9) and (10) therefore does not seem to deliver a result different from this one-line manipulation. If the intermediate approximations are in fact doing non-trivial work (e.g., bounding error terms, enabling the O(1/γ)O(1/\gamma) rate, or legitimizing the subsequent "drop both RAG terms" step), that should be stated explicitly; otherwise the derivation reads as padded, and the suspicion that it was generated or inflated by an AI is hard to dispel.

  2. The "adaptive to the retriever" claim rests on a single retriever. All experiments use E5-base-v2 as the base retriever, which makes it impossible to tell whether BRIDGE's gains generalize or whether the method is implicitly co-designed with E5's embedding space. The paper's central thesis — the retriever should first be optimized to make useful evidence available, and the LLM should then learn to search and reason with that evidence (line 49) — is a claim about retrievers in general, so it should be supported with a broader set of base retrievers (e.g., BGE, Contriever, GTE, or a sparse baseline such as BM25 combined with a dense reranker). At minimum, one additional dense retriever and one qualitatively different retriever (sparse or hybrid) should be reported on the open-domain QA suite, so that the order-sensitivity finding and the BRIDGE gains can be shown to transfer.

Questions for authors

see weakness

Reviewer_zinLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BRIDGE, a memory-efficient first-order bilevel framework for jointly optimizing a retriever and an LLM policy in agentic RAG. It combines RAG likelihood and outcome-based RL signals for retriever adaptation, while training the LLM through RL. The paper provides convergence analysis under idealized assumptions and reports encouraging improvements on open-domain and medical QA benchmarks.

Critical strengths
  • S1: The paper combines theoretical analysis with strong empirical results. It provides convergence analysis under stated assumptions and reports improvements across open-domain and medical QA benchmarks. These complementary results support the promise of jointly optimizing retrieval and policy learning.
Critical weaknesses
  • W1: The evaluation lacks direct retrieval metrics (e.g., recall or ndcg) and component-level comparisons, such as comparing the trained retriever with its initial version and alternatives like (e.g., bm25 or bge-m3) under a fixed LLM, or evaluating different LLMs with the same trained retriever. This makes it difficult to determine whether the gains come from better retrieval, improved LLM behavior, or their interaction.
Questions for authors
  • Q1: Do the retriever and LLM exhibit mismatched learning dynamics during joint training, and what evidence supports their stable convergence in practice?
Reviewer_4NwSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multiturn search agents that are trained for retrieval augments question answering. To answer a question, an agent can interact and issue multiple retrieval calls in a sequence to get to the required answer. The authors use the bilevel framework to design a setup where the retriever and the agent are co-trained. They first claim that there is a hierarchy in this optimization, and the retrieval needs to be optimized before the model is optimized to use the retrieval and not the other way around. Then they use a bilevel setup called BRIDGE, and based on the observation of the reward curvature make assumptions about the RL objective not changing too much near the retrieval’s RAG objective’s optimal parameters. The authors give a convergence guarantee under idealized setting but this discourse is mainly in the appendix so I did not read the result or the proof too deeply. The authors show the resulting method BRIDGE outperforms baselines by posttraining two Qwen models (2.3-3B and 2.4-7B) on QA and multihop QA benchmarks on average and is generally in the top 1 or 2.

The authors also run experiments on medical QA datasets as well and show improvements over baselines.

Critical strengths
  • The work gives an interesting perspective on the retrieval augment multi turn search agents by modeling it as a bilevel optimization problem. The resulting algorithm is a simple and intuitive alternative optimization approach, for the agent's objective, we just optimize it based on the current retriever and the RL objective and for the retrieval, we add the weighted sum of RL and RAG rewards.
  • The resulting algorithm consistently beats the baselines, the paper conducts tests on many QA baselines.
Critical weaknesses

A lot of assumptions that are used to justify the bilevel setup and thus the gains from it are not well supported or the numbers are close to standard errors.

  • RL then RAG and RAG then RL are compared within vt to highlight the hierarchy order in Figure 2, it seems the gap between the two is within the expected sampling errors that might come from the test set itself, without multiple seed runs, gains of such small percentages cannot be considered statistically better. Thus the motivation for the hierarchy is not experimentally supported correctly

  • I believe the arguments of dropping the last 2 terms in the Eq 9 is circular. It depends on the $ |\eta^* - \eta^*{\gamma} | $ being o(1/γ)o(1/ \gamma) is true when $\nabla\eta J_{{RL}} (\theta, \eta) = 0 $ .

  • In that case, at the optimal η∗\eta^*, i.e. the optimal point of the retriever of the decoupled rag should also be an optimal point of bridge. So shouldn’t the decoupled RAG and bridge have the same optimal solution? Then why use bridge?

  • The plots in some direction being flat does not really imply the optimal points will be close. The paper does implement the dual state solution which is the true implementation of the bilevel setup, but never reports its performance or actually visualizes the gap between the two retriever parameters in this case.

  • The asymptotic results where the error of the estimator is o(1/γ)o(1/ \gamma) is based on asymptotic values but the chosen γ\gamma is just 0.25

Questions for authors
  • Can you please report the standard errors for your tables?
  • Is it possible to add the results for the two estimator bilevel methods? What is the actual gap between the two retriever parameters in this case?
Reviewer_NAhSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of information credit gap in Agentic Reinforcement Learning where LLM’s are penalized even if the information retrieved by RAG is incorrect/missing. The authors propose a bilevel framework, BRIDGE, which considers retriever at lower level and LLM at higher level and trains both the retriever and the LLM. Specifically, BRIDGE first optimizes the retriever with RAG likelihood and multi turn GRPO loss and then updates the LLM parameters via mutli turn GRPO loss. Empirical results showcase higher gains for multi-hop QA than general QA across Qwen models.

Critical strengths
  • Simple idea: The paper combines the RAG likelihood (the likelihood of correct answer given the retrieved documents weighted by retireval probability of document) with RL to train the retriever.

  • Training of retriever requires less memory and time in comparison to baselines, thereby focusing on efficient training.

  • Comprehensive Experiments: The paper evaluates multiple benchmarks across different model backbones and model sizes. Further, the paper includes interesting ablations on different components, optimization order and compute allocation.

Critical weaknesses
  • Missing motivation for the work: No experiment showcases the different types of failures on a multi hop QA/general QA dataset. Specifically, what is the percentage of failures of missing evidence, misleading evidence and LLMs failure. How this distribution changes with BRIDGE? Specifically, do we observe any improvement in missing and misleading evidence?

  • No way to attribute the retriever's contribution from LLM's contribution: No experiments showcase if the retrieval quality actually improves. How does recall vary before and after the training. Further, it would be interesting to see how the BRIDGE's trained retriever when combined with VT-Search/Search R1 behaves? Does it improve the existing performance of these methods with the trained retriever? Adding these experiments will strengthen the overall motivation and empirical evaluation, which is currently missing.

  • Loss landscape plotted in Figure 3 and 5 showcases that RL loss is flat across different retriever directions (eta). This is expected as top-m documents are retrieved and the small perturbations barely change the information context. However, if RL loss doesn't depend on the retriever, then it contradicts the original motivation of the paper that retriever training is necessary to improve the final reward. The authors should provide a clarification and make sure that Figure 5a) to 5c) has correct axis labels mentioned.

  • For the component ablation in Table 3, it would be good to have only the retriever update without the LLM updates. The paper's idea of training retriever via RL+RAG likelihood can be used.

Minor Issues:

  • Fig. 1(b) maximizes J_RL while Eq. 1 and 6 minimize it. Mention it explicitly as it makes the reading difficult.
  • P for retriever is never reported. There should be an ablation over P as well.
  • Line 196 and 197 mentions k and m. These values seem to be missing in the paper.
Questions for authors
  • See the weaknesses above

  • Is Qwen3-Embedding-8B retriever being trained in Table 6? If yes, why the performance of Qwen3-8B and E5-base-v2 similar? Include these details in the paper.

  • Why order should matter: While the paper mentions that the order of retrieval and then RL matters. Does it really matter if we utilize the same rollouts for LLM update without changing the context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.