- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of automatic fact checking when the knowledge base is polluted by false information, and proposes the DEDUCE framework. This method decomposes complex statements into sub statements, uses the big language model to identify the support, refutation and irrelevant positions of retrieval evidence, dynamically updates the reliability of paragraphs according to the consistency of evidence and aggregation judgment, and feeds back the reliability to subsequent retrieval reordering. The author verifies the effectiveness of the method on seven datasets and four open source models. At the same time, through collaborative false information, mixed true and false evidence, and reputation poisoning experiments, the author analyzes the application boundaries of the consistency driven mechanism.
Critical strengths
-
The paper focuses on the fact verification when the knowledge base is polluted by false information, breaking through the common assumption that retrieval evidence is natural and reliable. The evaluation combined with directional synthetic pollution and natural false information is helpful to understand the performance of the fact verification system in an unreliable knowledge environment.
-
The method combines statement decomposition, evidence position aggregation, paragraph reliability update and retrieval reordering, so that historical verification information can affect subsequent evidence selection. This feedback mechanism provides a feasible idea for mitigating the interference of pollution evidence on multi-step verification.
-
The paper verifies the performance improvement on seven datasets and four open source models, and compares with multi class methods. The cooperation of false information, mixed true and false paragraphs and reputation poisoning experiments further reveal the applicable boundaries of the method.
Critical weaknesses
-
The method adjusts the weight according to the consistency of evidence and current aggregate judgment, but consensus consistency is not equal to fact correctness. When the initial judgment error or false evidence is highly consistent, the feedback mechanism may amplify the error, and the conditions for its effective operation and error recovery ability need to be analyzed.
-
The paper does not clearly give the fusion formula of reliability and retrieval relevance, the size of candidate pool and the score normalization method. The update extent directly depends on the aggregation score difference, so the retrieval score scale may affect the weight evolution, and the lack of relevant details will affect the reproduction and mechanism understanding.
-
Existing ablation does not completely isolate the role of aggregation weighting and retrieval feedback, and the relationship between different ablation configurations needs to be clarified. Compared with some strong baselines, the improvement is relatively small. It is suggested to supplement strictly matched control experiments and confidence intervals to enhance the persuasiveness of the core conclusions.
Questions for authors
1.How to fuse reliability and retrieval relevance for reordering? How to set the size of the candidate pool, the score normalization method, and the update range?
2.Under the same paragraph granularity, candidate evidence and position input, what results will be obtained by enabling aggregation weighting and retrieval feedback respectively? What are the differences between the "single" in Table 8 and the "no feedback" in Table 9?
3.How to divide calibration set and test set for frozen weight calibration experiment? Whether the overlapping of topics, entities and paragraphs is controlled, and whether the comparison method gets the same opportunity for calibration or online update?
4.Compared with TruthFinder and robust RAG methods, whether the performance improvement is significant under multiple independent runs or paired statistical tests?