acceptodds
Under review as a conference paper at ICLR 2027

AdaDR-RAG: Adaptive Agentic RAG Guided by Diversity and Reliability

Abstract

Early Retrieval-Augmented Generation (RAG) approaches typically adopt a static paradigm where models passively incorporate externally retrieved knowledge to generate answers. In contrast, Agentic RAG reframes question answering as a dynamic interaction between the model and its environment characterized by an alternating cycle of reasoning and search engine invocation. However, existing studies in Agentic RAG suffer from two primary limitations. First, they retrieve a fixed number of knowledge snippets at each interaction step regardless of information sufficiency. Second, they predominantly focus on final answer accuracy while neglecting the diversity and reliability of the iterative interaction trajectories. To address these challenges, we propose AdaDR-RAG, an Adaptive Diversity and Reliability-guided Agentic RAG framework. First, an adaptive search controller dynamically modulates retrieval depth with an exploration-exploitation balancing mechanism, ensuring sufficient evidence accumulation while preventing degenerate query loops and context pollution. Second, a trajectory clustering strategy extracts cluster-level rewards to jointly optimize for diversity and reliability, preventing policy collapse via sparsity-aware incentives and ensuring factual consistency through intra-group consensus to stabilize multi-step reasoning. Experimental results demonstrate that our method outperforms state-of-the-art baselines on widely-used benchmarks for both general and multi-hop question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Agentic RAG, Reinforcement Learning, Multi-Hop Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KVJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper proposes AdaDR-RAG, an agentic RAG framework trained with reinforcement learning. It claims that existing agentic RAG methods retrieve a fixed number of documents per step regardless of information sufficiency, and that they optimize only final-answer correctness while ignoring the diversity and reliability of the search trajectories, and proposes two solutions. First, retrieval returns one chunk per search step and the agent can repeat the same query to page to the next cached chunk, with a reward that keeps the repeat ratio inside a tolerance band. Second, rollouts for a question are clustered by the semantic similarity of their query sequences, and each trajectory receives a diversity reward inversely proportional to its cluster size and a reliability reward equal to its cluster's mean answer correctness, added to the standard exact-match reward. The proposed method is evaluated on seven benchmarks with ten baselines.

Critical strengths
  1. The design of the reward shaping is reasonable for addressing the claims raised by the authors.
  2. The ablation study is comprehensive.
Critical weaknesses
  1. The motivating claim in line 60, "These studies failed to account for exploration diversity and factual reliability." is a strong claim about prior methods and is not supported anywhere in the paper. No baseline is measured for policy collapse or for behavior on ambiguous queries, and no benchmark of ambiguous questions is used. The only related evidence, Figure 3 and Table 3, compares AdaDR-RAG with and without its own reward terms, which still does not show that prior methods suffer from the stated failure. The claim should either be supported with measurements on the baselines or softened.
  2. The contribution of each component is shown only as aggregate deltas, with no qualitative analysis of how the component changes the model's behavior. There is no example of a trajectory that RdivR_{div} diversifies, no example of a query strategy that RrelR_{rel} stabilizes, and no example of the paging mechanism retrieving evidence at depth that fixed-k misses. The two case studies in Appendix K compare the full model with Search-R1 and do not isolate any component.
  3. The baseline rows in Table 1 are reproduced digit-for-digit from other papers, and the evaluation protocols do not match. Those baseline rows match Search-P1's Table 1, which is labeled Accuracy, defined in that paper as whether the ground-truth string is contained in the response. All rows are presented under a single EM heading with no indication that any number is cited. The results are therefore not comparable.
Questions for authors
  1. What evidence supports the claim in line 60 that prior methods fail on exploration diversity and factual reliability? Can the authors show the attributed collapse on the existing methods under the same protocol?
  2. Can the authors give qualitative examples isolating each component: a strategy that appears only with RdivR_{div}, a strategy that RrelR_{rel} stabilizes, and a question where paging reaches evidence that fixed-k misses?
  3. Which rows in Table 1 were run by the authors and which were cited? For the cited rows, what is the evaluation protocol? Are they measured with EM or with accuracy?
3 comments

I thank the authors for the responses. In Response 3, the clarification that the reported metric is reference-answer containment raises a new concern. The manuscript states "We use Exact Match (EM)," and the TreePS-RAG and Search-R1 numbers in Table 1 are strict exact match by those papers' definitions and released code, while the Search-P1 and HiPRAG numbers are containment. The table therefore mixes two scoring rules. Could the authors provide a version of Table 1 in which all rows are reported under a single metric? Otherwise the results are not comparable.

Thank you for the re-scored results. However, Search-R1's paper states "Exact Match (EM) is used as the evaluation metric," and its evaluation script calls em_check, not subem_check. Could the authors point out where the Search-R1 codebase uses subem_check to compute the reported results? Otherwise the five rows taken from Search-R1 remain strict EM, and Table 1 still mixes two scoring rules.

I thank the authors for the response. However, my concerns for Q2 and Q3 are not addressed. For Q2, no qualitative examples were provided. For Q3, the function compute_score_subem cited in the response is not used anywhere in the Search-R1 codebase, and Search-P1's Search-R1 results are copied from Search-R1. Therefore, Table 1 still mixes two scoring rules.

Reviewer_awe7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AdaDR-RAG, an agentic retrieval-augmented generation framework combining query-aware adaptive search with diversity- and reliability-guided reinforcement learning. The search controller pages through cached results when consecutive search queries match and starts a new retrieval otherwise. A trajectory-level balancing reward discourages excessive repetition and rapid query switching. Query trajectories are clustered through stepwise semantic similarity, and cluster-size-based diversity rewards and cluster-level answer-correctness rewards are combined with terminal accuracy and balancing rewards in GRPO. Experiments on seven general and multi-hop QA datasets using Qwen2.5-3B/7B show improved mean exact-match accuracy relative to strong baselines. The paper also reports ablations and additional evidence on retrieval efficiency, training-seed stability, and supporting facts.

Critical strengths
  1. The paper addresses a practical and well-motivated issue: fixed document counts per agent search action may waste context or provide insufficient evidence. Its cache-based continuation controller is relatively simple to implement and has potential operational value.

  2. The empirical evaluation is reasonably broad. Besides seven QA benchmarks and two model sizes, the authors include module ablations, fixed-k comparisons, three-seed analysis, supporting-fact scores, retrieval-efficiency measurements, and retriever-transfer experiments.

Critical weaknesses
  1. The proposed reliability reward is defined as the within-cluster fraction of correct final answers (Eq. 5), not as a direct measure of evidence credibility, faithful reasoning, or factual support. As such, it may largely smooth and redistribute outcome supervision. The paper's evidence for improved QA accuracy does not fully establish the stronger claim of improved factual reliability, which is central to the contribution.

  2. The contribution of the adaptive control rule is not fully isolated from finer retrieval granularity, caching, document-injection budget, and the learned policy. The proposed method injects one cached chunk per search action, while key RL-agent baselines use multiple chunks per action (Appendix H.1). Existing fixed-k and w/o-Ada tests are helpful but do not fully resolve these interacting factors under identical evidence and compute budgets. Also, exact repeated-query matching is an indirect behavioral proxy rather than an explicit test of information sufficiency.

Questions for authors
  1. Can the authors explain what the within-cluster answer-accuracy reward captures beyond conventional terminal outcome supervision? Please provide a direct assessment of factual grounding, evidence faithfulness, or consistency that disentangles these constructs from final-answer EM.

  2. Can the authors compare the adaptive controller with fixed or alternative controllers under matched search-call counts, retrieved/injected documents, context-token budgets, and training settings, to isolate the benefit of the depth decision itself?

  3. How are historical trajectory buffers bounded and maintained as the policy evolves? In particular, how are historical answer-correctness labels stored or updated and used when computing current cluster rewards? Please clarify possible staleness effects and the additional clustering/training cost.

Reviewer_58GnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaDR-RAG, an agentic RAG method that adjusts retrieval depth and encourages diverse search strategies. The agent can repeat a query to obtain more cached passages or issue a new query. The method groups similar search trajectories and uses cluster size and average answer correctness as additional training rewards. Experiments on seven QA benchmarks show improvements over several agentic RAG baselines.

Critical strengths

The paper addresses a practical problem: when an agent should retrieve more evidence and when it should change its search direction. The method is straightforward and does not require an external verifier during training.

The experiments are fairly comprehensive. Besides answer accuracy, the paper reports results across three training seeds, supporting-fact retrieval, inference efficiency, and domain transfer. The improvements over strong baselines are encouraging.

Critical weaknesses

The reliability reward averages answer correctness within each cluster but does not directly check whether the reasoning is supported by retrieved evidence. It is unclear how much it adds beyond giving more weight to answer correctness.

The historical trajectory buffer needs more analysis. The paper does not explain how outdated trajectories are handled or report the additional training time and memory costs. A comparison with using only current rollouts is also missing.

The retrieval comparison does not fully separate the effects of pagination and reward design. AdaDR-RAG receives one passage per action, while the main baselines receive three. Although the paper includes fixed-k experiments, comparing the methods with the same pagination interface would make the gains more convincing.

Questions for authors

How does the reliability reward compare with increasing the accuracy reward weight? Does it improve evidence grounding when the other components are unchanged?

How is the historical buffer maintained as training progresses? What happens to performance and training cost when only current rollouts are used?

Would the gains remain if the main baselines used the same cached pagination interface?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.