- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Tree of Evidence, or ToE, for automated fact-checking. Instead of asking an LLM to directly judge a claim, ToE searches for evidence from several sources and organises the evidence in a tree. It explicitly searches for both supporting and contradicting evidence. When the evidence is not considered reliable enough, the system breaks the claim into smaller subclaims and searches again.
The system contains three main parts: an RL-based multi-source retrieval agent, a learned evidence evaluator, and a tree aggregation model. The authors also provide a theoretical analysis of the retrieval process.
The method is evaluated on LIAR, PolitiFact, Check-COVID, and an adversarial dataset called AdvFact. It is compared with direct prompting and several existing fact-checking approaches using two LLM backbones. The reported results show that ToE is often better than the compared methods, with the largest improvement appearing on AdvFact with DeepSeek-V3.2. However, it does not outperform the strongest baseline in every dataset and backbone combination.
Critical strengths
-
The paper addresses an important and timely problem. Fact-checking systems that retrieve information from the web can be misled when fabricated content is deliberately made easy to retrieve. The paper directly studies this problem and proposes a structured defence rather than relying only on the LLM’s internal knowledge.
-
The explicit search for counter-evidence and the evidence-tree structure are useful ideas. ToE does not search only for evidence supporting the claim. It also deliberately searches for contradicting evidence, which helps reduce one-sided retrieval. The evidence tree also preserves subclaims, evidence, and intermediate scores, making the result easier to inspect than a direct LLM verdict. The paper’s retrieval ablation suggests that removing counter-evidence search substantially reduces performance, although this result is based on a small evaluation set.
Critical weaknesses
- The experiments do not justify the full complexity of the proposed pipeline. ToE adds many components at the same time: RL-based source selection, multiple retrieval tools, LLM query generation, document cleaning, source and stance analysis, learned reliability estimation, claim decomposition, neural tree aggregation, pruning, and stopping rules.
The reported ablations mainly remove groups of retrieval sources. They do not separately show the value of the main architectural components, such as the learned retrieval policy, claim decomposition, learned reliability score, neural tree aggregation, or pruning. A simpler baseline using the same LLM, retrieval sources, retrieved-document budget, and evidence-processing steps would help determine whether the tree and RL components provide gains beyond additional retrieval and inference.
- The adversarial evaluation is difficult to interpret because all AdvFact examples are false. The paper states that AdvFact contains 200 examples and that all are labelled false. If ordinary accuracy is calculated only on this dataset, a trivial system that always predicts false would score 100%. This makes the reported AdvFact accuracy insufficient by itself to demonstrate robustness to poisoning.
A more convincing evaluation would include both true and false claims, matched clean and poisoned examples, an always-false baseline, attack success rate, and the change in performance from clean to poisoned evidence. This concern is important because AdvFact provides a central part of the evidence supporting the paper’s poisoning-robustness claim.
Questions for authors
-
Clarification and additional evidence: Can the authors provide separate ablations for the learned retrieval policy, claim decomposition, learned reliability estimation, neural tree aggregation, and pruning? These results would materially affect whether I view the full architecture as justified.
-
Clarification and additional comparison: How would ToE compare with a simpler pipeline using the same LLM backbone, retrieval sources, number of retrieved documents, and evidence extraction, but using a fixed retrieval strategy and transparent weighted aggregation instead of RL and a tree?
-
Clarification: How should the AdvFact accuracy be interpreted when all examples are false? What accuracy would an always-false baseline achieve under the reported evaluation? Can the authors report results using both true and false claims, or matched clean and poisoned versions of the same claims?
-
Clarification: How much inference does ToE use relative to the baselines? At minimum, could the authors report the average number of searches, documents, LLM calls, tokens, and tree nodes per claim? It is needed to separate the effect of the proposed method from the effect of greater inference-time computation.
Thank you for the response. I increased my review score.