acceptodds
Under review as a conference paper at ICLR 2027

Structured Evidence Filtering and Consensus Reasoning for Multi-Agent Fact-Checking

Abstract

Large-scale retrieval-augmented fact-checking often suffers from noisy evidence and unreliable multi-agent reasoning aggregation. To address these challenges, we propose a Structured Evidence Filtering-based Multi-Agent Consensus Framework (), consisting of two core modules. The first is a Structured Evidence Filtering (SEF) module, which leverages domain knowledge to remove irrelevant evidence and maintain semantic consistency with the target claim. The second is a Challenge-and-Validation Reasoning Approach, which treats minority dissent as a signal to activate iterative cross-examination, allowing agents to correct logical defects through evidence-grounded deliberation. Experiments on three benchmarks—CheckThat-T3, CHEF, and SciFact—show that SEF-MAC consistently outperforms competitive baselines, with Micro-F1 gains of up to 3.72 percentage points and Macro-F1 gains of up to 3.00 percentage points. Further ablation and analysis suggest that evidence filtering and consensus reasoning make complementary contributions to improve fact-checking accuracy across different evidence settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fact‑Checking, Multi‑Agent Systems, Retrieval‑Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LMrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the SEF-MAC framework is proposed to solve the problem of evidence noise and unreliable multi-agent judgment in retrieval enhanced fact checking. Structured evidence filtering module evaluates evidence from four dimensions: core fact relevance, time consistency, entity alignment and subject coherence, and filters through dynamic threshold. Then, the three verification agents with different roles make independent judgments. When there are differences, they can start up to two rounds of questioning and verification to form the final conclusion. On CheckThat-T3, CHEF and SciFact, the author uses two large models to verify the detection effect, and analyzes the role of evidence filtering and bifurcation driven reasoning.

Critical strengths
  1. The paper deals with the evidence noise at the input end and the judgment divergence at the reasoning end respectively to avoid relying only on increasing the number of agents to improve performance. The results of ablation showed that filtering and challenge verification played complementary roles in different evidence environments.

  2. The author not only compares the final indicators, but also analyzes the initial divergent samples and the majority voting performance. On CheckThat-T3, the improvement over the initial voting is supported by paired self-service sampling and McNemar inspection to further verify the disputed samples.

  3. The evaluation covers Chinese and English, general fields and scientific statements, and uses two model backbones. The paper also compares different reasoning strategies, the number of debate rounds and the filtering weight, which provides rich information for the application scope of the understanding method.

Critical weaknesses
  1. Scoring rules give low scores to time conflicts and entity mismatches, but these information may also be the key to refuting false statements. For example, when the evidence corrects the event time or participants, "inconsistency" with the statement does not mean lack of verification value. It is suggested to clearly distinguish the evidence pointing to different events and the evidence revealing the statement error, and report the retention rate of supporting evidence and refuting evidence respectively.

  2. The text says that 283 statements have corresponding summaries. Table 5 also reports that the total number of original evidences is 283, and 269 after filtering. If this means that each non empty evidence pool has only one summary, its standard deviation is zero, and the dynamic threshold is equal to the evidence score. The rule of "score not lower than threshold" should not be deleted; The minimum rule should also be retained. The author needs to clarify the statistical unit of the abstract and fragment, the processing of empty evidence and the actual filtering process, otherwise the relevant ablation results are difficult to interpret.

  3. In case of continuous divergence, the method selects the agent that references the most fragments, but the number of references cannot directly represent the quality of evidence, independence or correctness of reasoning, and may also favor lengthy references. The existing comparison also involves the division of roles, additional reasoning and arbitration rules, which makes it difficult to isolate their contributions. It is suggested that the arbitration methods based on the validity of reference should be melted separately under the same call or token budget.

Questions for authors
  1. When the evidence clearly corrects the time, entity or value in the statement, how can SEF avoid deleting it as "inconsistent evidence"? Is there any manually marked counterevidence retention rate and filtering error analysis?

  2. Why are the 283 original evidences of SciFact reduced to 269 after filtering? These numbers correspond to summary, fragment or non empty declaration respectively. Does the actual implementation contain filter conditions not described in the body?

  3. What is the arbitration accuracy rate of "the largest number of references" in the continuous disagreement sample? How will the results change after changing to majority voting, fixed roles or citing quality scores?

Reviewer_1KPgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEF-MAC, a training free fact checking framework combining structured evidence filtering with disagreement triggered multi-agent deliberation. The filtering module scores candidate snippets on factual relevance, temporal consistency, entity alignment, and thematic coherence. Three role specialized agents then independently verify the claim, with non-unanimous predictions triggering up to two debate rounds. Experiments on three datasets report improvements across two backbone models. The paper also provides prompts, component ablations, debate-round analysis, and paired significance testing. These are useful contributions, but I have concerns about the filtering implementation, preservation of refuting evidence, attribution of the debate gains, and evaluation protocols.

Critical strengths
  1. In the field of fact checking, noisy retrieved evidence and unreliable aggregation are meaningful problems. Selectively allocating additional reasoning to initially disputed cases is a sensible design.

  2. The paper reports disagreement rates, role-specific prediction distributions, convergence rates, and accuracy on debated cases. These analyses are more informative than aggregate classification results alone.

  3. The CheckThat-T3 comparison against initial voting includes paired bootstrap intervals and McNemar’s test. The reported 200 corrected versus 104 newly introduced errors establish a net improvement for that configuration.

  4. The appendix provides prompts, scoring rubrics, decoding settings, and call-count analysis. The equal-weight experiment also appropriately shows that the exact SEF weights are not essential to the reported CHEF result.

Critical weaknesses
  1. The filtering rubric may conflate irrelevant mismatches with informative contradictions. The temporal consistency score assigns zero to a temporal contradiction. However, in fact checking, a temporal contradiction can be the decisive reason to refute a claim. Evidence stating that the same event occurred in a different year is potentially highly useful, whereas evidence concerning an unrelated event or time period is not. The current rubric does not clearly distinguish these cases. Similar ambiguity arises from repeatedly describing the retained set as “claim-consistent.” Evidence utility should be stance-neutral: both supporting and refuting evidence can be highly useful.

The case study does retain refuting snippets, so I am not claiming that SEF necessarily removes all counterevidence. Rather, the concern is that its scoring rule may systematically penalize some important forms of refutation. Please evaluate retention separately for supporting, refuting, and irrelevant evidence, ideally including targeted temporal and numerical contradictions. Downstream accuracy and a shift toward particular source categories do not establish that critical counterevidence is preserved.

  1. The contribution of challenge-and-validation is confounded with the final arbitration rule. For unresolved cases, the system selects the agent citing the most retained snippets, breaking ties in favor of the Contradiction Detection Agent. Citation quantity is not equivalent to evidential support: an agent can cite multiple irrelevant or redundant snippets, while a correct decision may require only one decisive source. More importantly, the comparison against initial majority voting changes both deliberation and final aggregation. On CheckThat-T3, 111 cases remain unresolved after debate and are decided by the fallback.

  2. The main text states that CheckThat-T3 inference uses the original True/False/Conflicting labels and only renames them for reporting. Appendix A.3 instead describes normalization before evaluation, while the shared prompts use Support/Refute/Neutral and parsing failures default to NEI. Conflicting evidence and insufficient evidence are not interchangeable.

  3. There is a numerical error in the case study: the scores in Table 14 imply a threshold of approximately 0.645 using population standard deviation, rather than 0.55. The retained set is unchanged in that example, but the discrepancy should be corrected.

Questions for authors
  1. How much improvement remains when the no-debate baseline uses the same final arbitration rule?

  2. Which label definitions and prompts were actually used for CheckThat-T3, particularly for Conflicting versus NEI?

  3. How can SciFact retain fewer snippets than the number of initially nonempty evidence pools under the stated threshold and fallback?

  4. How does SEF distinguish a genuinely refuting temporal contradiction from an irrelevant temporal mismatch?

Reviewer_DGqrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

SEF-MAC is a framework for retrieval-augmented fact-checking. Its first component scores each retrieved snippet based on core fact relevance, temporal consistency, entity alignment, and thematic coherence. Each of these four aspects is assigned a weight that reflects their relative importance. Snippets scored below a threshold are dropped from the retrieval results. Then, three agents with separate roles assess the snippets separately. These agents focus on evidence sufficiency, contradiction detection, and logical consistency, respectively. If they disagree, it triggers a debate between them which could last for up to two rounds. Ties are considered to be broken in favor of the agent citing the most snippets. The authors evaluated their pipeline on CheckThat-T3, CHEF and SciFact, and considered gpt-4o-mini and doubao-seed-2.0 as backbone models.

Critical strengths
  1. The debate is cost-aware which is logical. It runs only when its seems to be genuinely needed. This helps reducing the required LLM calls per claim.

  2. Various empirical aspects of the work show strengths. The mechanism and its components are analyzed carefully. The paper's evaluation setup covers diverse settings that include a variety of domains for claims. Each module is also ablated separately, and SEF is tested with both direct and CoT prompting. Furthermore, it appears it could be relatively easy to reproduce the work, as the method is fully specified, and all prompts, scoring rubrics, decoding settings and tie-breaking rules are provided.

Critical weaknesses
  1. The framework doesn't demonstrate much novelty. The ideas of triggering a debate only when it is needed and role-based debate for fact-checking have been both studied before (e.g., "Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning", arXiv:2504.05047; MADR, "Can LLMs Produce Faithful Explanations for Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate”, arXiv:2402.07401; De-bateCV). The paper does not compare its design with these, so it is unclear what it contributes beyond combining known components.

  2. How evaluation is done seems to be different from the benchmarks' standard setup. Instead of providing the evidence with CheckThat-T3, the authors search Google for the claim text without restricting the date. As the claims typically come from fact-checking websites, the retrieved results might include the published verdict itself. The paper's case study includes a snippet that states the answer directly. This makes it hard to tell how much of the performance comes from verification rather than from finding the verdict. This also makes the results incomparable with prior work on these benchmarks.

  3. SEF is only compared with using no filter, not with simpler filters such as a single relevance score or a standard reranker.

Questions for authors
  1. How does SEF affect each class? In particular, how often does it remove evidence that contradicts the claim, and how does Refute recall change with and without SEF?

  2. How often do the Google results contain a published fact-check or verdict for the claim? How do the results change with the official CheckThat-T3 evidence, or with retrieval restricted to before the claim's date?

  3. How much do the results vary across repeated runs? Can you report variance or confidence intervals for the main comparisons in Table 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.