- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposed a agentic RAG named Trust-RAG, which aims to address reasoning errors in large language models (LLMs) during multi-hop question answering errors that arise when the models retrieve irrelevant, unreliable, or contradictory evidence. The framework consists of three independent module: retrieval, verification, and consistency agents. It also leverage a consensus graph to deal with multi-documents with content contradiction, and also maintain the traceability of data sources, avoid to put the contradiction data into LLMs. It also maintain a Dynamic Retry Mechanism, if the consistency agent judge that the evidence is not enough to answer the question, the system will optimize query and retrieval again, to ensure the information consistency. The experimental performance achieved an EM of 0.423 and an F1 of 0.535 on the HotpotQA (multi-hop question answering) validation set, and an EM of 0.830 on the GSM8K (mathematical reasoning) test set.
Critical strengths
The paper have several advantages as follows:
- The proposed method improve the reliability and consistency of evidence.
- The proposed method processes a high degree of adaptability and self-correction capability.
- Multi-agent systems possess a clear framework for the division of labor.
Critical weaknesses
The Trust-RAG framework proposed in this study has the following drawbacks and limitations:
- Compromised coverage of complete evidence: To ensure that the information ultimately fed into the generator is free of contradictions, the system discards conflicting reasoning paths.
- Lack of ablation studies on the independent contribution of the module: The "contradiction handling" mechanism in the paper is evaluated within the context of the complete system pipeline; the reported data reflect overall performance but do not isolate the specific contribution of this mechanism on its own.
- Parameters and thresholds lack true dynamic flexibility: although the system claims to feature "adaptive" retrieval depth, it is currently restricted to a set of fixed options.
- Higher Computational Overhead: Evidence verification, the construction of consistency graphs, and the conditional re-retrieval of information within the system all introduce additional computational burdens.
Questions for authors
The whole paper seems like a AI generated paper.