acceptodds
Under review as a conference paper at ICLR 2027

Trust-RAG: Agentic Retrieval-Augmented Generation with Evidence Verification and Consistency-Aware Reasoning

Abstract

Retrieval-augmented generation (RAG) improves language models by providing relevant external information during answer generation. However, retrieved ev- idence can be low-quality or contradictory. When unreliable evidence is passed directly to the language model, it can lead to incorrect reasoning and halluci- nated answers. Trust-RAG is proposed as an agentic RAG framework designed to improve evidence reliability and consistency in multi-hop question answering. Trust-RAG consists of three agents: Retrieval, Verification, and Consistency. This is coordinated by an LLM-based controller. The Retrieval Agent adaptively se- lects the query and retrieval depth, the Verification Agent evaluates and filters retrieved documents, and the Consistency Agent identifies contradiction-free ev- idence and determines whether additional retrieval is required. The controller coordinates these decisions and can initiate re-retrieval when the available evi- dence is insufficient. The framework combines BM25 and dense retrieval to form a hybrid retrieval system, verifies evidence using relevance and NLI signals, ap- plies consistency-graph reasoning, and uses Qwen2.5-7B-Instruct for decision- making and answer generation. The system evaluates Trust-RAG on HotPotQA and GSM8K. On the full HotpotQA distractor validation set (n = 7,405), Trust- RAG achieves an Exact Match of 0.423 and an F1 score of 0.535, with selected evidence covering the gold supporting documents in 58.7% of examples. On GSM8K, Trust-RAG achieves a numerical Exact Match of 0.830 on the full test set (n = 1,319).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, Retrieval-Augmented Generation, Multi-Agent Systems, Evidence Verification, Consistency-Aware Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3biEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposed a agentic RAG named Trust-RAG, which aims to address reasoning errors in large language models (LLMs) during multi-hop question answering errors that arise when the models retrieve irrelevant, unreliable, or contradictory evidence. The framework consists of three independent module: retrieval, verification, and consistency agents. It also leverage a consensus graph to deal with multi-documents with content contradiction, and also maintain the traceability of data sources, avoid to put the contradiction data into LLMs. It also maintain a Dynamic Retry Mechanism, if the consistency agent judge that the evidence is not enough to answer the question, the system will optimize query and retrieval again, to ensure the information consistency. The experimental performance achieved an EM of 0.423 and an F1 of 0.535 on the HotpotQA (multi-hop question answering) validation set, and an EM of 0.830 on the GSM8K (mathematical reasoning) test set.

Critical strengths

The paper have several advantages as follows:

  1. The proposed method improve the reliability and consistency of evidence.
  2. The proposed method processes a high degree of adaptability and self-correction capability.
  3. Multi-agent systems possess a clear framework for the division of labor.
Critical weaknesses

The Trust-RAG framework proposed in this study has the following drawbacks and limitations:

  1. Compromised coverage of complete evidence: To ensure that the information ultimately fed into the generator is free of contradictions, the system discards conflicting reasoning paths.
  2. Lack of ablation studies on the independent contribution of the module: The "contradiction handling" mechanism in the paper is evaluated within the context of the complete system pipeline; the reported data reflect overall performance but do not isolate the specific contribution of this mechanism on its own.
  3. Parameters and thresholds lack true dynamic flexibility: although the system claims to feature "adaptive" retrieval depth, it is currently restricted to a set of fixed options.
  4. Higher Computational Overhead: Evidence verification, the construction of consistency graphs, and the conditional re-retrieval of information within the system all introduce additional computational burdens.
Questions for authors

The whole paper seems like a AI generated paper.

Reviewer_s5LTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Trust-RAG coordinates retrieval, document verification, and claim-consistency agents with an LLM controller. BM25 and dense retrieval are fused, retrieval depth is selected from 5/7/9, documents are filtered by relevance/reliability/contradiction scores, and a claim/entity graph selects consistent evidence and a reasoning path. Insufficient evidence can trigger up to two retries. Qwen2.5-7B-Instruct supplies agent decisions and final answers. Evaluation uses the full HotpotQA distractor validation set and GSM8K test set, with GSM8K training worked examples serving as its retrieval corpus. Reported scores are HotpotQA EM/F1 0.423/0.535 and GSM8K numerical EM 0.830. All gold supporting titles survive in 58.7% of HotpotQA evidence sets and 22.2% of selected paths (Table 2, Figures 2–3, pp.7–8).

Critical strengths
  1. Coherent and inspectable system decomposition. The distinction between document verification, claim-level conflict resolution, and final reading makes the intended evidence flow clear. The paper specifies retrieval models, several thresholds, bounded retrieval depths, and a retry budget; it also explicitly describes generation after budget exhaustion (§§2–3, pp.2–6).
  2. Useful scope and failure diagnostics. Evaluation covers both complete stated splits, with tuning described on a separate HotpotQA training subset (§3.1). Reporting complete gold-title coverage separately for the broader evidence set and selected path is informative. The authors candidly acknowledge limited path coverage, proxy measures rather than direct hallucination labels, and the lack of isolated contradiction-handling evaluation (§§4.3–4.6, pp.6–8).
Critical weaknesses
  1. The contribution beyond the underlying reader and retrieval is unestablished. Table 1 compares only fixed retrievers on Title Recall@7; Table 2 reports only the full system's answers. There is no same-Qwen answer baseline using fixed hybrid context, no no-retrieval baseline for GSM8K, and no ablation separating adaptive retrieval, verification, graph selection, or retries (§3.2, pp.5–7). Thus the results cannot distinguish benefits of Trust-RAG's new decisions from ordinary retrieval or the reader's existing reasoning ability. I credit the paper's modest applicability claim and its explicit deferral of ablations (§5), but they leave the proposed methodology's value unresolved. Targeted comparisons on the existing datasets, holding the reader and relevant evidence/token budgets comparable, would address this concern without expanding the project to new domains.
  2. The central trust/consistency mechanism is insufficiently defined and validated. The heuristic reliability term, document-level contradiction aggregation, path objective/search, and LLM override/sufficiency policies are not specified sufficiently to reconstruct the main decisions (§§2.3–2.6, Figure 1, §3.3). More fundamentally, removing the lower-confidence member of a detected conflict does not establish that the surviving claim is correct. Gold-title retention and answer-copy coincidences do not measure contradiction-detection accuracy or whether conflict resolution preserves the correct evidence (§§4.3–4.5). The reported 58.7% evidence coverage is candid and useful, but cannot demonstrate improved reliability without a comparator or direct conflict analysis. A precise mechanism plus targeted examples/measurements of detected conflicts and resolution outcomes would make this central claim assessable.
Questions for authors
  1. Is there an existing end-to-end comparison between Trust-RAG and fixed hybrid retrieval using the same Qwen2.5-7B-Instruct reader under comparable context and inference budgets that isolates the added value of the proposed agentic mechanisms?
Reviewer_qXq4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Trust-RAG, a retrieval-augmented generation (RAG) framework for multi-hop question answering. A controller coordinates three LLM-based stages: a Retrieval Agent that combines BM25 and dense retrieval and picks a retrieval depth, a Verification Agent that scores and filters documents, and a Consistency Agent that builds a claim-level graph, removes contradictory claims, and decides whether to trigger another round of retrieval. The authors evaluate the proposed method on the HotpotQA and GSM8K datasets without comparison to any baselines.

Critical strengths
  1. The problem of verifying the evidence in a RAG system is well-motivated.
Critical weaknesses
  1. The paper addresses a well-motivated problem, but in its current form it does not yet meet the standard expected of a conference paper. The main concerns are the presentation, the limited justification of the method design, and the absence of baselines in the experiments, each detailed below.
  2. The presentation could be improved. Figure 1 carries method details that do not appear in the text, so presenting the controller loop as pseudocode would make the method clearer and easier to reproduce. Related work is currently discussed within the Introduction, and a separate section would help position the contribution. Section 2 has seven subsections, several quite short, and mixes implementation details with the method description. Consolidating these would improve readability.
  3. The design choices would benefit from further justification. It is not clear why each component is needed, and no ablation is provided to show its contribution. Moreover, the confidence score in Section 2.4 uses fixed coefficients (0.5, 0.2, 0.3) and thresholds (0.05, 0.30, 0.60) without an explanation of how they were chosen or how sensitive the results are to them.
  4. The experiments do not include baselines for the full system. Table 2 reports Trust-RAG only, so comparisons with LLM without retrieval, a standard RAG, and related methods cited in the paper would be needed to assess the claimed improvements in answer quality and evidence reliability.
Questions for authors
  1. Could the authors clarify what they consider the main contribution of this work, and why it is significant relative to existing approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.