acceptodds
Under review as a conference paper at ICLR 2027

Grounded-R1: Reinforcement Learning for Preemptive Hallucination Intervention and Grounded Generation

Abstract

Hallucination in large language models (LLMs) often begins locally: an ungrounded claim can enter the generation trajectory, steer subsequent reasoning, and compound into a global failure. Retrieval-augmented generation (RAG) can reduce such failures by supplying external evidence, but to prevent local error propagation, evidence must arrive before risky claims propagate. However, existing methods do not directly learn retrieval trigger timing from local hallucination risk. Retrieval is driven either by heuristic rules or fixed procedures in training-free methods or by delayed, coarse indirect rewards in reinforcement learning (RL)-based approaches, causing evidence to arrive too late to prevent local error propagation while unnecessary calls waste the retrieval budget. In this paper, we propose Grounded-R1, a dynamic RAG framework for preemptive hallucination intervention. Grounded-R1 introduces the Real-time Hallucination Risk Detection (RHRD) module to estimate sentence-level hallucination risk from generation-time signals and uses two-phase RL to align retrieval trigger timing with this risk. Across eight knowledge-intensive benchmarks, Grounded-R1 improves final-answer exact match (EM) by 18.0% over the strongest external baseline with comparable retrieval calls. On five open-ended benchmarks, it raises sentence-level non-hallucination rate (NHR) by 12.7 points. Delivering the same evidence one sentence earlier reduces next-sentence hallucination rate by 11.1 points; under equal retrieval budgets, Grounded-R1 targets positions with an observed 8.3-point downstream hallucination reduction. Grounded-R1 thus improves grounded generation not by retrieving more, but by retrieving where one sentence of delay matters. Our code and models are available at https://sites.google.com/view/grounded-r1.

Primary area:
reinforcement learning
Keywords:
retrieval-augmented generation, large language models, hallucination mitigation, reinforcement learning, dynamic retrieval, grounded generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qu11OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination in LLM generation, focusing on when external retrieval should be triggered to prevent errors from propagating. The authors propose Grounded-R1, a dynamic RAG framework that combines Real-time Hallucination Risk Detection (RHRD) with two-phase reinforcement learning. RHRD estimates sentence-level hallucination risk using generation-time signals. The first training phase optimizes answer quality and retrieval cost, while the second uses sentence-level risk-based rewards to improve retrieval timing. Experiments on eight knowledge-intensive benchmarks compare Grounded-R1 with existing RAG methods in terms of answer quality, hallucination rate, and retrieval efficiency.

Critical strengths

(i) The paper focuses on hallucinations in knowledge-intensive LLM generation, where unsupported claims can affect subsequent reasoning and lead to incorrect final answers.

(ii) The experimental evaluation is extensive, covering eight knowledge-intensive benchmarks and comparisons with various RAG methods. The paper also includes ablation studies, retrieval-cost analysis, sentence-level hallucination evaluation, and controlled experiments examining the effect of retrieval timing.

Critical weaknesses

(i) The paper does not establish whether hallucination risk is a suitable signal for deciding when retrieval is beneficial. Eq. (6) rewards retrieval when the RHRD score is high, while Appendix B.9 states that the sentence-level reward does not estimate the counterfactual downstream utility of retrieval. The paper evaluates RHRD against hallucination labels and uses intervention experiments to show that earlier retrieval can improve subsequent generation. However, these experiments do not examine whether positions with higher hallucination risk are also those where retrieval provides greater benefits.

(ii) The paper does not establish why its sentence-end retrieval decisions can better address premature or delayed retrieval than existing methods. Approaches such as FLARE and IRCoT already use sentence-level generation or reasoning steps to guide retrieval. Section 3.2 explains the choice of sentence-end decisions as a trade-off between timeliness and semantic completeness, but does not show why this choice leads to better retrieval timing. Table 4 demonstrates that inserting the same evidence one sentence earlier improves subsequent generation, rather than showing that the proposed decision mechanism is better at choosing when to retrieve. Although Table 5 compares retrieval-position selection under equal budgets, the direct differences between Grounded-R1 and the compared policies are not statistically conclusive in Appendix J.7. Therefore, the claimed advantage in avoiding premature or delayed retrieval over existing approaches remains uncertain.

Questions for authors

(i) Please provide further analysis of the relationship between RHRD scores and the actual downstream benefits of retrieval.

(ii) Please provide further evidence that sentence-end decisions help avoid premature or delayed retrieval compared with existing approaches.

Reviewer_wuNaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces GROUNDED-R1, a new RL approach for RAG to more explicitly teach RAG systems to effectively trigger retrieval when the risks of hallucinations are high. Existing methods trigger retrieval based on heuristics (eg FLARE) or train adaptive RAG systems using RL (eg Search-R1) with outcome-level rewards. GROUNDED-R1 conducts a two-stage RL to directly learn retrieval trigger timing from local hallucination risks. Specifically, after phase 1 RL, which uses standard answer reward, format reward, and retrieval cost penalty, phase 2 training introduces a new step-level reward based on the new RHRD score. RHRD score extends RIND in DRAGIN (Su et al., 2024). If the RHRD score is high, the policy should trigger retrieval; if it is low, it should continue thinking or answer. Experimental results show strong performance on both extractive, short-form QA and open-ended generation.

Critical strengths
  • This work studies an important problem and proposes a well-motivated, technically sound method.
  • Results across multiple benchmarks are strong.
  • This paper is easy to follow and presents results and analysis clearly.
Critical weaknesses

The biggest weakness is the relatively limited technical contribution. As acknowledged by the authors, RHRD is pretty similar to RIND, and providing denser feedback for RAG systems beyond outcome rewards has been studied in prior work such as Zhang et al., 2025. While using two-stage RL based on whether retrieval is triggered at the sentence level is, IIRC, new, I believe several additional ablations are required to identify the core components. I believe the authors should provide

  1. Ablations of RHRD - what happens if they use RIND in phase 2 RL?
  2. Effectiveness of Phase 2 RL beyond prolonged training - what happens if they continue RL for the same number of steps as the current Phase 2 but using the original reward for Phase 1?

Additionally, I think comparing with methods that introduce denser rewards for RAG such as Zhang et al., 2025. would strengthen the paper.

Questions for authors

I listed two questions and potential additional ablations in weaknesses.

Reviewer_jnnROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Grounded-R1 trains a Search-R1-style agent to decide at each sentence end whether to retrieve. Phase I optimizes an episode-level reward combining an LLM-judge answer score with retrieval-cost penalties. Phase II adds a sentence-level reward based on RHRD, a risk score adapted from DRAGIN's RIND (entropy × attention × content-word × subword count), used only in training. With Qwen2.5-7B, the paper reports +18% relative macro EM over Search-R1 across eight QA benchmarks at about 2 retrievals per question, and +12.7 pp sentence-level non-hallucination rate. Controlled interventions show that inserting the same evidence one sentence earlier reduces next-sentence hallucination by 11.1 pp.

Critical strengths

S1. Carefully controlled timing intervention. Immediate and Delayed conditions receive byte-identical retrieval blocks, and the judge sees only evidence available before each sentence (App. I). This gives RHRD-independent evidence that timing matters (−11.1 pp, Table 4).

S2. Equal-budget selector analysis, candidly reported. All selectors rank a shared pool of 2,000 positions under fixed budgets, and the authors openly note where differences are not significant (Table 40), which builds trust.

S3. RHRD as a training signal rather than a trigger. The learned policy outperforms direct threshold gating while using half as many calls (2.04 vs 4.12; Table 2), suggesting it learns more than threshold imitation.

Critical weaknesses

W1. Novelty relative to step-reward search agents is under-differentiated. At inference, Grounded-R1 runs exactly like Search-R1, so any difference comes from the training reward and the extra training steps. LeTS (Zhang et al., EMNLP 2025; cited but not compared) already mixes step-level and outcome rewards. StepSearch (EMNLP 2025), IKEA (Huang et al., 2025), and Search Wisely (Wu et al., EMNLP 2025) target step-wise or over-/under-retrieval rewards but are not discussed.

W2. The Search-R1 baseline appears under-reproduced, which may account for much of the headline gain. Since the difference from Search-R1 is small, the clarification is even more important here:

The Search-R1 row in Table 1 is the authors' own retraining (Table 21), and it falls well below the numbers Jin et al. (COLM 2025, Table 3) report for Qwen2.5-7B-Instruct. On Bamboogle, which uses the identical 125-question set, it scores 0.208 vs. 0.400 (GRPO) and 0.368 (PPO). Across the six benchmarks both papers report, Grounded-R1 (full) averages 0.367, below the published Search-R1 averages of 0.391 (GRPO) and 0.384 (PPO). Subsets and evaluation settings differ, but the gap is too large to be sampling noise. Other choices further cloud the margins: SeaKR scores EM = 0.000 on four datasets, and the "LLM" metric reuses the training-reward judge. W3. The evidence does not yet show that Phase II teaches better retrieval timing.

Under PPO, where Phase I is healthy, the Phase I reward adds +2.1 macro EM and Phase II adds +1.1, each in a single run (Table 43). The large GRPO gain (+17.9) mostly reflects recovery from under-converged Phase I checkpoints (Table 44). Furthermore, risk is scored on a sentence that has already been written, and search follows it (e.g., the "2004" claim in Fig. 2). The method therefore limits error propagation rather than preempting errors. The matched-prefix study shows that earlier evidence helps, but the policy's choice of positions is not significantly better than Search-R1's (+1.0 [−10.3, +12.8], Table 40). An otherwise matched Phase II run with , with multiple seeds, would settle this. W4. I'm concerned that the sentence-wise decision will increase the latency. Please report per-query inference latency (wall-clock) and generated tokens for Grounded-R1, Search-R1, the hard-threshold control, FLARE and DRAGIN under the same serving setup.

Questions for authors

Address concerns in weakness session.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.