- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the SEF-MAC framework is proposed to solve the problem of evidence noise and unreliable multi-agent judgment in retrieval enhanced fact checking. Structured evidence filtering module evaluates evidence from four dimensions: core fact relevance, time consistency, entity alignment and subject coherence, and filters through dynamic threshold. Then, the three verification agents with different roles make independent judgments. When there are differences, they can start up to two rounds of questioning and verification to form the final conclusion. On CheckThat-T3, CHEF and SciFact, the author uses two large models to verify the detection effect, and analyzes the role of evidence filtering and bifurcation driven reasoning.
Critical strengths
-
The paper deals with the evidence noise at the input end and the judgment divergence at the reasoning end respectively to avoid relying only on increasing the number of agents to improve performance. The results of ablation showed that filtering and challenge verification played complementary roles in different evidence environments.
-
The author not only compares the final indicators, but also analyzes the initial divergent samples and the majority voting performance. On CheckThat-T3, the improvement over the initial voting is supported by paired self-service sampling and McNemar inspection to further verify the disputed samples.
-
The evaluation covers Chinese and English, general fields and scientific statements, and uses two model backbones. The paper also compares different reasoning strategies, the number of debate rounds and the filtering weight, which provides rich information for the application scope of the understanding method.
Critical weaknesses
-
Scoring rules give low scores to time conflicts and entity mismatches, but these information may also be the key to refuting false statements. For example, when the evidence corrects the event time or participants, "inconsistency" with the statement does not mean lack of verification value. It is suggested to clearly distinguish the evidence pointing to different events and the evidence revealing the statement error, and report the retention rate of supporting evidence and refuting evidence respectively.
-
The text says that 283 statements have corresponding summaries. Table 5 also reports that the total number of original evidences is 283, and 269 after filtering. If this means that each non empty evidence pool has only one summary, its standard deviation is zero, and the dynamic threshold is equal to the evidence score. The rule of "score not lower than threshold" should not be deleted; The minimum rule should also be retained. The author needs to clarify the statistical unit of the abstract and fragment, the processing of empty evidence and the actual filtering process, otherwise the relevant ablation results are difficult to interpret.
-
In case of continuous divergence, the method selects the agent that references the most fragments, but the number of references cannot directly represent the quality of evidence, independence or correctness of reasoning, and may also favor lengthy references. The existing comparison also involves the division of roles, additional reasoning and arbitration rules, which makes it difficult to isolate their contributions. It is suggested that the arbitration methods based on the validity of reference should be melted separately under the same call or token budget.
Questions for authors
-
When the evidence clearly corrects the time, entity or value in the statement, how can SEF avoid deleting it as "inconsistent evidence"? Is there any manually marked counterevidence retention rate and filtering error analysis?
-
Why are the 283 original evidences of SciFact reduced to 269 after filtering? These numbers correspond to summary, fragment or non empty declaration respectively. Does the actual implementation contain filter conditions not described in the body?
-
What is the arbitration accuracy rate of "the largest number of references" in the continuous disagreement sample? How will the results change after changing to majority voting, fixed roles or citing quality scores?