- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a two-stage framework for evidence-grounded multimodal fact-checking. Specifically, the authors observed two limitations of the current fact-checking systems: (1) existing systems reason primarily about verification operations rather than the verification strategy; (2) a well-planned investigation may retrieve evidence that is relevant but insufficient. Built upon these findings, they design a new framework consisting with two steps: (1) generating multiple claim-specific verification strategies; (2) emotion-guided evidence-gap repair. Empirically, the proposed framework achieves good performance on four textual and multimodal fact-checking benchmarks.
Critical strengths
-
Illustrative figures (e.g., Figure 1 and Figure 2) help the audience to understand the core idea.
-
Extensive related work (>1 page) describes the background and motivation.
Critical weaknesses
-
The experimental analysis is not deep enough. From the reading experience, the authors generally report the performance scores.
-
In Appendix A, the authors report the efficiency results, where RETRACE increased response latency and other costs. In this case, I doubt the trade-off behind this paper.
-
Formatting: (1) Line 138 misses a dot “.”. (2) For consistency, GPT-4o and the series in Table 2 should also include a citation using “\citep{}”. (3) The leading phrase in each paragraph has a dot in line 368, but drops the dot from line 375 onward. Please keep the consistency.
Questions for authors
- Why are there many unavailable cells (“-”) in Table 2?
5 comments
Response to Reviewer zc5w (1)-Continued
- Quantitative Failure-Mode Analysis
We further extend the qualitative failure analysis by quantifying the five previously identified failure modes across 350 randomly sampled incorrectly classified instances from the four benchmarks. The analysis was conducted by a graduate-level researcher, with each instance assigned to one primary, mutually exclusive failure category.
The final verifier accounts for most errors (65.4%), mainly through Label Ambiguity, where RETRACE retrieves largely adequate evidence but chooses a neighbouring label (e.g., NEI vs. Refuted, or OOC vs. Miscaptioned). Stage II and Stage I account for 21.7% and 10.0%, respectively, and only 2.9% arise from unavailable external sources. Fine-grained label discrimination in the final verifier is therefore the main target for future improvement, followed by more complete gap repair.
| Failure Mode | Stage | Samples | Percentage |
|---|---|---|---|
| Label Ambiguity | Final Verification | 174 | 49.7% |
| Incomplete Gap Repair | Stage II | 76 | 21.7% |
| Verification Error | Final Verification | 55 | 15.7% |
| Strategy–Task Mismatch | Stage I | 35 | 10.0% |
| Evidence Access Failure | External Retrieval | 10 | 2.9% |
- Consistency of Final Verification Predictions
We ran RETRACE three times on 400 claims (100 randomly sampled per benchmark). Pairwise agreement, the fraction of run pairs per claim that assign the same label, is 91.4%. A three-run majority vote therefore gives a definite label for 99.75% of claims, and claims where runs disagree (12.75%) can be flagged for human review, which supports auditable deployment.
| Prediction Consistency | Samples | Percentage |
|---|---|---|
| All three runs agree | 349 | 87.25% |
| Two runs agree; one differs | 50 | 12.50% |
| All three runs disagree | 1 | 0.25% |
- Consistency of Stage I Verification-Strategy Selection
We further examine whether Stage I consistently selects similar verification strategies when processing the same input across repeated executions. To ensure a consistent assessment of Stage I verification strategies, we define five verification-strategy categories based on their primary investigative objectives: (1) Textual Fact/Source Verification, which examines factual claims and the credibility of their originating sources; (2) Image–Text Consistency, which assesses whether visual content supports or contradicts the accompanying textual claim; (3) Visual Authenticity, which investigates potential image manipulation or fabrication; (4) Time/Place/Entity Verification, which checks temporal, geographical, and entity-related information; and (5) Cross-Source Corroboration, which retrieves relevant external evidence and verifies claims by comparing information across multiple independent sources. Since Stage I generates free-form verification strategies, we categorize them according to their primary verification objective. Two strategies are considered similar when they belong to the same predefined category, even if their natural-language descriptions differ.
As shown in the below table, the same category was selected in all three runs for 53.25% of claims and in at least two runs for 84.75%. Stage I therefore varies more than the final labels do. However, 77.1% (47 out of 61) of claims with three different strategies still received the same label in all runs, which shows that the later stages (evidence acquisition, gap assessment and targeted repair) compensate for variation in the initial strategy.
| Strategy Consistency | Samples | Percentage |
|---|---|---|
| All three runs select the same or similar strategies | 213 | 53.25% |
| Two runs select the same or similar strategies | 126 | 31.50% |
| All three runs select different strategies | 61 | 15.25% |
- Stage II Prediction-Transition Analysis
We examined Stage II using a separate set of 400 claims (100 per benchmark), comparing the preliminary prediction after Stage I with the final prediction after Stage II. All claims undergo evidence-gap diagnosis, but only 85 (21.25%) trigger additional retrieval.
| Benchmark | Correct → Correct | Incorrect → Correct | Correct → Incorrect | Incorrect → Incorrect |
|---|---|---|---|---|
| AVeriTeC | 81 | 12 | 2 | 5 |
| MOCHEG | 84 | 10 | 1 | 5 |
| VERITE | 82 | 12 | 3 | 3 |
| MMFakeBench | 83 | 9 | 1 | 7 |
| Total | 330 | 43 | 7 | 20 |
Overall, 43 of 63 initially incorrect predictions (68.3%) are corrected, while 330 of 337 initially correct predictions (97.9%) remain correct, with seven regressions. Accuracy increases from 84.25% to 93.25% (+9.0 percentage points), with gains of 10.0, 9.0, 9.0, and 8.0 points across the four benchmarks, respectively. However, these results may not represent the full benchmark performance distributions.
Response to Reviewer zc5w (2)
Critical Weakness 2: In Appendix A, the authors report the efficiency results, where RETRACE increased response latency and other costs. In this case, I doubt the trade-off behind this paper.
Thank you for raising this important concern regarding the computational overhead of RETRACE and whether its verification improvements justify the additional inference cost. We acknowledge that RETRACE requires more computational resources than direct GPT-5.2 prompting. As reported in Appendix A (Table 5), RETRACE requires an average latency of 4 minutes 31 seconds and an API cost of USD 0.18 per claim on VERITE, compared with 10 seconds and USD 0.01 for direct GPT-5.2, corresponding to approximately 27 times higher latency and 18 times higher API cost. However, this additional expenditure is accompanied by substantial accuracy improvements of 16.9, 23.8, 30.3, and 45.3 percentage points on VERITE, AVeriTeC, MMFakeBench, and MOCHEG, respectively. Importantly, unlike direct prompting, RETRACE performs structured verification-strategy discovery, external evidence acquisition, and targeted evidence-gap repair, enabling evidence-supported verdicts rather than relying solely on the model's parametric knowledge. Moreover, RETRACE's API expenditure of less than USD 0.20 per claim remains substantially below reported costs associated with human-led verification. For context, one study adopted an estimated cost of €80 per professional fact-check [1], while other study reported USD 7.50 per manually annotated claim [2]. Although these figures represent different verification tasks and RETRACE's API cost excludes human oversight and broader deployment expenses, they illustrate the potential economic advantage of automated evidence-driven verification. Furthermore, check-worthiness filtering, as discussed in Appendix A, offers an additional opportunity to reduce unnecessary verification expenditure in high-volume applications.
To further examine the accuracy–efficiency trade-off, we conducted an additional three-objective Pareto comparison of RETRACE with DEFAME, MMD-Agent, and T²Agent, considering verification accuracy, average inference latency, and API cost per claim. We focus on MMFakeBench because it provides a common evaluation benchmark for all four frameworks. DEFAME and MMD-Agent were rerun using GPT-5.2 as the common reasoning backbone, with comparable retrieval budgets and evaluation samples. For T²Agent, we retain the 75.3% accuracy reported in its original publication and additionally evaluate 100 samples using GPT 5.2 to estimate its average inference latency and API cost. The results are summarized in the below table.
| Method | Accuracy (%) | Latency (s) | API Cost (USD) | Pareto-efficient |
|---|---|---|---|---|
| RETRACE | 84.2 | 259.0 | 0.19 | Yes |
| DEFAME | 71.4 | 264.0 | 0.15 | Yes |
| MMD-Agent | 64.7 | 55.3 | 0.38 | Yes |
| T²Agent | 75.3 | 272.4 | 1.68 | No |
As shown in the Table, RETRACE achieves the highest accuracy of 84.2%, with an average latency of 259.0 seconds and an API cost of USD 0.19 per claim. Compared with DEFAME, RETRACE improves accuracy by 12.8 percentage points while reducing latency by 5.0 seconds, at an additional API cost of USD 0.04. Compared with MMD-Agent, RETRACE achieves 19.5 percentage points higher accuracy and USD 0.19 lower API cost per claim, although MMD-Agent provides substantially lower latency (55.3 seconds). Furthermore, RETRACE outperforms T²Agent by 8.9 percentage points, reduces latency by 13.4 seconds, and decreases the estimated API cost by approximately 88.7%. The resulting Pareto analysis identifies RETRACE, DEFAME, and MMD-Agent as non-dominated configurations, while T²Agent is dominated by RETRACE under the reported measurements. Nevertheless, differences in evaluation settings and the use of separately obtained T²Agent measurements should be considered when interpreting the Pareto comparison, while RETRACE, DEFAME, and MMD-Agent were evaluated under comparable conditions on the same MMFakeBench samples.
References:
-
Cagé, J., Gallo, N., Hengel, M., Henry, E., & Huang, Y. (2025). Fact-checking and misinformation: Evidence from the market leader.
-
Niwa, A., Yamada, M., Nakayama, K., Kaneko, M., Suzuki, H., & Sekine, S. (2026). Rethinking the Evaluation of LLM Fact-Checkers. arXiv preprint arXiv:2610.30504.
Response to Reviewer zc5w (1)
We would like to thank the reviewer for this comment. Beyond the main performance scores, we added six analyses that explain how and where RETRACE succeeds or fails:
We first investigate how RETRACE selects different verification strategies across the four benchmarks. As shown in the below table, the distribution of selected strategies varies with the characteristics of each benchmark. On AVeriTeC, textual fact and source verification accounts for 71.4% of the selected strategies. In contrast, evidence retrieval and corroboration dominate on MOCHEG (78.6%) and VERITE (72.3%), reflecting the importance of retrieving supporting evidence and assessing the contextual authenticity of claims. On MMFakeBench, textual fact and source verification (51.7%) and image–text consistency assessment (38.6%) represent the two most frequently selected strategies.
These differences suggest that Stage I adapts its verification priorities to the characteristics of different misinformation benchmarks rather than relying on a uniform investigation strategy. In particular, the substantial proportion of image–text consistency strategies on MMFakeBench highlights the importance of cross-modal correspondence for verifying potentially misleading image–text combinations. This analysis provides additional evidence of the task-dependent behavior of RETRACE's strategy-discovery mechanism.
Using the confusion matrices in the original manuscript (Appendix Tables), we computed the relative error reduction per class against direct GPT-5.2 and GPT-5.2 CoT: RER = (E_baseline − E_RETRACE) / E_baseline. Positive values mean fewer errors.
The results demonstrate that RETRACE reduces errors in 11 of 13 classes, by 37.5–85.6%. They include every class on MOCHEG (74.4–80.4%) and MMFakeBench (54.1–85.3%).
On VERITE, RETRACE shows a favourable trade-off: direct GPT-5.2 tends to label true claims as false, and RETRACE corrects 182 of these errors (244 → 62) at the cost of 13 additional False-class errors (58 → 71), giving 169 fewer errors in total.
On AVeriTeC, the Conflicting class remains difficult, with RETRACE classifying 8 of 38 claims correctly, compared with 10 for GPT-5.2 and 13 for CoT; Recognising that conflicting evidence is therefore an important direction for future work.