acceptodds
Under review as a conference paper at ICLR 2027

RETRACE: REasoning through Verification sTRAtegies and Emotion-Guided EvidenCE Repair for Multimodal Fact-Checking

Abstract

Effective multimodal fact-checking requires more than retrieving topically relevant information: a system must determine which investigative route is most likely to yield decisive evidence and recognize when the retrieved evidence leaves important aspects of a claim unresolved. We present RETRACE (REasoning through verification sTRAtegies and emotion-guided evidenCE repair), a dual-stage adaptive framework for evidence-grounded multimodal fact-checking. RETRACE first generates multiple claim-specific verification strategies representing alternative paths to decisive evidence and ranks them through pairwise preference modeling. The selected strategy is then translated into prioritized, executable forensic tasks and carried out with multimodal tools to construct an initial evidence state. After observing the initial evidence, it diagnoses unresolved evidential gaps by jointly assessing factual evidence coverage with textual, visual and cross-modal emotional signals to repair the evidence state. It initiates a targeted second investigation aimed at recovering the missing or insufficient evidence. An independent evidence-grounded verifier then determines the final claim label from the evidence state. Without task-specific fine-tuning, RETRACE achieves accuracies of 80.8% on AVeriTeC, 87.1% on MOCHEG, 86.7% on VERITE, and 84.2% on MMFakeBench, improving over the strongest reported comparison on each benchmark by 2.8, 25.9, 2.8, and 8.9 percentage points, respectively. An ablation study further demonstrates that verification strategy discovery and emotion-guided evidence repair provide complementary benefits to fact-checking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Fact-Checking, Agentic Reasoning and Planning, Evidence-Grounded Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zc5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage framework for evidence-grounded multimodal fact-checking. Specifically, the authors observed two limitations of the current fact-checking systems: (1) existing systems reason primarily about verification operations rather than the verification strategy; (2) a well-planned investigation may retrieve evidence that is relevant but insufficient. Built upon these findings, they design a new framework consisting with two steps: (1) generating multiple claim-specific verification strategies; (2) emotion-guided evidence-gap repair. Empirically, the proposed framework achieves good performance on four textual and multimodal fact-checking benchmarks.

Critical strengths
  • Illustrative figures (e.g., Figure 1 and Figure 2) help the audience to understand the core idea.

  • Extensive related work (>1 page) describes the background and motivation.

Critical weaknesses
  • The experimental analysis is not deep enough. From the reading experience, the authors generally report the performance scores.

  • In Appendix A, the authors report the efficiency results, where RETRACE increased response latency and other costs. In this case, I doubt the trade-off behind this paper.

  • Formatting: (1) Line 138 misses a dot “.”. (2) For consistency, GPT-4o and the series in Table 2 should also include a citation using “\citep{}”. (3) The leading phrase in each paragraph has a dot in line 368, but drops the dot from line 375 onward. Please keep the consistency.

Questions for authors
  • Why are there many unavailable cells (“-”) in Table 2?
5 comments
AuthorsOpenReview ↗

Response to Reviewer zc5w (1)

Critical Weaknesses 1: The experimental analysis is not deep enough. From the reading experience, the authors generally report the performance scores.

We would like to thank the reviewer for this comment. Beyond the main performance scores, we added six analyses that explain how and where RETRACE succeeds or fails:

AnalysisMain Insight
1. Stage I strategy distributionStrategy choice adapts to each benchmark's type of misinformation.
2. Class-wise error reductionErrors fall in 11 of 13 classes; VERITE shows a favourable trade-off (169 fewer errors overall), while the rare Conflicting class remains hard.
3. Failure-mode analysis65% of errors arise in the final verifier, mostly between neighbouring labels.
4. Prediction consistency87% of claims receive the same label in all three runs.
5. Strategy consistencyStrategies vary more than final labels (53% full agreement).
6. Stage II prediction transitionsStage II corrects 68% of initially wrong predictions while changing only 2% of correct ones.
  1. Analysis of Stage I Verification-Strategy Distribution

We first investigate how RETRACE selects different verification strategies across the four benchmarks. As shown in the below table, the distribution of selected strategies varies with the characteristics of each benchmark. On AVeriTeC, textual fact and source verification accounts for 71.4% of the selected strategies. In contrast, evidence retrieval and corroboration dominate on MOCHEG (78.6%) and VERITE (72.3%), reflecting the importance of retrieving supporting evidence and assessing the contextual authenticity of claims. On MMFakeBench, textual fact and source verification (51.7%) and image–text consistency assessment (38.6%) represent the two most frequently selected strategies.

These differences suggest that Stage I adapts its verification priorities to the characteristics of different misinformation benchmarks rather than relying on a uniform investigation strategy. In particular, the substantial proportion of image–text consistency strategies on MMFakeBench highlights the importance of cross-modal correspondence for verifying potentially misleading image–text combinations. This analysis provides additional evidence of the task-dependent behavior of RETRACE's strategy-discovery mechanism.

Broad Strategy CategoryAVeriTeCMOCHEGVERITEMMFakeBench
Textual Fact & Source Verification71.417.00.051.7
Image–Text Semantic Consistency0.23.227.638.6
Visual Authenticity & Plausibility0.00.00.04.4
Time, Place & Entity Grounding11.01.30.12.0
Evidence Retrieval & Cross-Source Corroboration17.478.672.33.2
  1. Class-Wise Error Reduction Analysis

Using the confusion matrices in the original manuscript (Appendix Tables), we computed the relative error reduction per class against direct GPT-5.2 and GPT-5.2 CoT: RER = (E_baseline − E_RETRACE) / E_baseline. Positive values mean fewer errors.

DatasetClassRETRACE vs. GPT-5.2 (%)RETRACE vs. GPT-5.2 CoT (%)
AVeriTeCSupported+69.8+59.6
NEI+37.5+52.4
Refuted+65.7+68.9
Conflicting−7.1−20.0
MMFakeBenchOriginal+85.3+85.6
Textual Distortion+58.3+70.6
Mismatch+54.1+38.6
Visual Distortion+66.7+57.7
VERITETrue+74.6+72.3
False−22.4−20.3
MOCHEGSupported+80.4+80.1
NEI+78.6+77.8
Refuted+74.4+73.3

The results demonstrate that RETRACE reduces errors in 11 of 13 classes, by 37.5–85.6%. They include every class on MOCHEG (74.4–80.4%) and MMFakeBench (54.1–85.3%).

On VERITE, RETRACE shows a favourable trade-off: direct GPT-5.2 tends to label true claims as false, and RETRACE corrects 182 of these errors (244 → 62) at the cost of 13 additional False-class errors (58 → 71), giving 169 fewer errors in total.

On AVeriTeC, the Conflicting class remains difficult, with RETRACE classifying 8 of 38 claims correctly, compared with 10 for GPT-5.2 and 13 for CoT; Recognising that conflicting evidence is therefore an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer zc5w (1)-Continued

  1. Quantitative Failure-Mode Analysis

We further extend the qualitative failure analysis by quantifying the five previously identified failure modes across 350 randomly sampled incorrectly classified instances from the four benchmarks. The analysis was conducted by a graduate-level researcher, with each instance assigned to one primary, mutually exclusive failure category.

The final verifier accounts for most errors (65.4%), mainly through Label Ambiguity, where RETRACE retrieves largely adequate evidence but chooses a neighbouring label (e.g., NEI vs. Refuted, or OOC vs. Miscaptioned). Stage II and Stage I account for 21.7% and 10.0%, respectively, and only 2.9% arise from unavailable external sources. Fine-grained label discrimination in the final verifier is therefore the main target for future improvement, followed by more complete gap repair.

Failure ModeStageSamplesPercentage
Label AmbiguityFinal Verification17449.7%
Incomplete Gap RepairStage II7621.7%
Verification ErrorFinal Verification5515.7%
Strategy–Task MismatchStage I3510.0%
Evidence Access FailureExternal Retrieval102.9%
  1. Consistency of Final Verification Predictions

We ran RETRACE three times on 400 claims (100 randomly sampled per benchmark). Pairwise agreement, the fraction of run pairs per claim that assign the same label, is 91.4%. A three-run majority vote therefore gives a definite label for 99.75% of claims, and claims where runs disagree (12.75%) can be flagged for human review, which supports auditable deployment.

Prediction ConsistencySamplesPercentage
All three runs agree34987.25%
Two runs agree; one differs5012.50%
All three runs disagree10.25%
  1. Consistency of Stage I Verification-Strategy Selection

We further examine whether Stage I consistently selects similar verification strategies when processing the same input across repeated executions. To ensure a consistent assessment of Stage I verification strategies, we define five verification-strategy categories based on their primary investigative objectives: (1) Textual Fact/Source Verification, which examines factual claims and the credibility of their originating sources; (2) Image–Text Consistency, which assesses whether visual content supports or contradicts the accompanying textual claim; (3) Visual Authenticity, which investigates potential image manipulation or fabrication; (4) Time/Place/Entity Verification, which checks temporal, geographical, and entity-related information; and (5) Cross-Source Corroboration, which retrieves relevant external evidence and verifies claims by comparing information across multiple independent sources. Since Stage I generates free-form verification strategies, we categorize them according to their primary verification objective. Two strategies are considered similar when they belong to the same predefined category, even if their natural-language descriptions differ.

As shown in the below table, the same category was selected in all three runs for 53.25% of claims and in at least two runs for 84.75%. Stage I therefore varies more than the final labels do. However, 77.1% (47 out of 61) of claims with three different strategies still received the same label in all runs, which shows that the later stages (evidence acquisition, gap assessment and targeted repair) compensate for variation in the initial strategy.

Strategy ConsistencySamplesPercentage
All three runs select the same or similar strategies21353.25%
Two runs select the same or similar strategies12631.50%
All three runs select different strategies6115.25%
  1. Stage II Prediction-Transition Analysis

We examined Stage II using a separate set of 400 claims (100 per benchmark), comparing the preliminary prediction after Stage I with the final prediction after Stage II. All claims undergo evidence-gap diagnosis, but only 85 (21.25%) trigger additional retrieval.

BenchmarkCorrect → CorrectIncorrect → CorrectCorrect → IncorrectIncorrect → Incorrect
AVeriTeC811225
MOCHEG841015
VERITE821233
MMFakeBench83917
Total33043720

Overall, 43 of 63 initially incorrect predictions (68.3%) are corrected, while 330 of 337 initially correct predictions (97.9%) remain correct, with seven regressions. Accuracy increases from 84.25% to 93.25% (+9.0 percentage points), with gains of 10.0, 9.0, 9.0, and 8.0 points across the four benchmarks, respectively. However, these results may not represent the full benchmark performance distributions.

AuthorsOpenReview ↗

Response to Reviewer zc5w (2)

Critical Weakness 2: In Appendix A, the authors report the efficiency results, where RETRACE increased response latency and other costs. In this case, I doubt the trade-off behind this paper.

Thank you for raising this important concern regarding the computational overhead of RETRACE and whether its verification improvements justify the additional inference cost. We acknowledge that RETRACE requires more computational resources than direct GPT-5.2 prompting. As reported in Appendix A (Table 5), RETRACE requires an average latency of 4 minutes 31 seconds and an API cost of USD 0.18 per claim on VERITE, compared with 10 seconds and USD 0.01 for direct GPT-5.2, corresponding to approximately 27 times higher latency and 18 times higher API cost. However, this additional expenditure is accompanied by substantial accuracy improvements of 16.9, 23.8, 30.3, and 45.3 percentage points on VERITE, AVeriTeC, MMFakeBench, and MOCHEG, respectively. Importantly, unlike direct prompting, RETRACE performs structured verification-strategy discovery, external evidence acquisition, and targeted evidence-gap repair, enabling evidence-supported verdicts rather than relying solely on the model's parametric knowledge. Moreover, RETRACE's API expenditure of less than USD 0.20 per claim remains substantially below reported costs associated with human-led verification. For context, one study adopted an estimated cost of €80 per professional fact-check [1], while other study reported USD 7.50 per manually annotated claim [2]. Although these figures represent different verification tasks and RETRACE's API cost excludes human oversight and broader deployment expenses, they illustrate the potential economic advantage of automated evidence-driven verification. Furthermore, check-worthiness filtering, as discussed in Appendix A, offers an additional opportunity to reduce unnecessary verification expenditure in high-volume applications.

To further examine the accuracy–efficiency trade-off, we conducted an additional three-objective Pareto comparison of RETRACE with DEFAME, MMD-Agent, and T²Agent, considering verification accuracy, average inference latency, and API cost per claim. We focus on MMFakeBench because it provides a common evaluation benchmark for all four frameworks. DEFAME and MMD-Agent were rerun using GPT-5.2 as the common reasoning backbone, with comparable retrieval budgets and evaluation samples. For T²Agent, we retain the 75.3% accuracy reported in its original publication and additionally evaluate 100 samples using GPT 5.2 to estimate its average inference latency and API cost. The results are summarized in the below table.

MethodAccuracy (%)Latency (s)API Cost (USD)Pareto-efficient
RETRACE84.2259.00.19Yes
DEFAME71.4264.00.15Yes
MMD-Agent64.755.30.38Yes
T²Agent75.3272.41.68No

As shown in the Table, RETRACE achieves the highest accuracy of 84.2%, with an average latency of 259.0 seconds and an API cost of USD 0.19 per claim. Compared with DEFAME, RETRACE improves accuracy by 12.8 percentage points while reducing latency by 5.0 seconds, at an additional API cost of USD 0.04. Compared with MMD-Agent, RETRACE achieves 19.5 percentage points higher accuracy and USD 0.19 lower API cost per claim, although MMD-Agent provides substantially lower latency (55.3 seconds). Furthermore, RETRACE outperforms T²Agent by 8.9 percentage points, reduces latency by 13.4 seconds, and decreases the estimated API cost by approximately 88.7%. The resulting Pareto analysis identifies RETRACE, DEFAME, and MMD-Agent as non-dominated configurations, while T²Agent is dominated by RETRACE under the reported measurements. Nevertheless, differences in evaluation settings and the use of separately obtained T²Agent measurements should be considered when interpreting the Pareto comparison, while RETRACE, DEFAME, and MMD-Agent were evaluated under comparable conditions on the same MMFakeBench samples.

References:

  1. Cagé, J., Gallo, N., Hengel, M., Henry, E., & Huang, Y. (2025). Fact-checking and misinformation: Evidence from the market leader.

  2. Niwa, A., Yamada, M., Nakayama, K., Kaneko, M., Suzuki, H., & Sekine, S. (2026). Rethinking the Evaluation of LLM Fact-Checkers. arXiv preprint arXiv:2610.30504.

Reviewer_pCp7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RETRACE, a two-stage framework for multimodal fact-checking. It first generates and ranks verification strategies to guide evidence collection, then combines factual coverage and emotional signals to identify gaps and trigger targeted retrieval. A separate verifier predicts the final label from the collected evidence. Experiments across four benchmarks report improvements over prior systems, with ablations examining strategy discovery and evidence repair.

Critical strengths

Using emotional signals to guide evidence acquisition is reasonable and interesting. Emotional framing may highlight aspects of a claim that deserve further investigation. The paper uses these signals to guide retrieval rather than treating them as evidence of truth or falsehood. This is a sensible distinction. The repair ablation also provides encouraging evidence that emotional signals can complement factual coverage.

Critical weaknesses

The main comparisons do not adequately isolate the effectiveness of the proposed method. RETRACE uses GPT-5.2, while Table 2 does not establish that prior systems were rerun with the same backbone, tools, retrieval access, and evaluation samples. The reported improvements therefore mix methodological changes with differences in the underlying systems. Without controlled comparisons against strong prior approaches, it remains unclear how much of the gain comes from RETRACE’s specific design.

The main comparisons do not adequately isolate the effectiveness of the proposed method. RETRACE uses GPT-5.2, while Table 2 does not establish that prior systems were rerun with the same backbone, tools, retrieval access, and evaluation samples. The reported improvements therefore mix methodological changes with differences in the underlying systems. Without controlled comparisons against strong prior approaches, it remains unclear how much of the gain comes from RETRACE’s specific design.

Questions for authors
  1. Were the prior methods in Table 2 rerun under a shared setup, or were their scores taken from the original papers? Are the evaluation samples and evidence conditions identical?

  2. Can you provide comparisons against strong prior systems using the same backbone and comparable retrieval and inference budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pCp7 (1)

Critical Weaknesses 1: The main comparisons do not adequately isolate the effectiveness of the proposed method. RETRACE uses GPT-5.2, while Table 2 does not establish that prior systems were rerun with the same backbone, tools, retrieval access, and evaluation samples. The reported improvements therefore mix methodological changes with differences in the underlying systems. Without controlled comparisons against strong prior approaches, it remains unclear how much of the gain comes from RETRACE’s specific design.

Critical Weaknesses 2: The main comparisons do not adequately isolate the effectiveness of the proposed method. RETRACE uses GPT-5.2, while Table 2 does not establish that prior systems were rerun with the same backbone, tools, retrieval access, and evaluation samples. The reported improvements therefore mix methodological changes with differences in the underlying systems. Without controlled comparisons against strong prior approaches, it remains unclear how much of the gain comes from RETRACE’s specific design.

We thank the reviewer for raising this important concern regarding the fairness of comparisons with existing methods and the need to isolate the contribution of RETRACE's proposed verification framework.

To directly address this concern, we selected two established multimodal fact-checking frameworks, DEFAME and MMD-Agent, for additional evaluations against RETRACE. We reran DEFAME using GPT-5.2 as the common reasoning backbone across AVeriTeC, MOCHEG, VERITE, and MMFakeBench, with identical evaluation samples, consistent label mappings, and a maximum budget of 10 web-search queries per claim. Furthermore, we disabled DEFAME's default exclusion of professional fact-checking websites to allow unrestricted web retrieval consistent with RETRACE. We additionally reproduced MMD-Agent using GPT-5.2 on MMFakeBench, the benchmark for which it was originally designed, with the same 10-query retrieval limit and Serper.dev Google Search API used by RETRACE. The reproduced DEFAME and MMD-Agent results are explicitly identified as “Reproduced” in Table 2 and are identical to those reported in the controlled comparison in Appendix H. The results of these evaluations are summarized in the following table.

DatasetRETRACE (GPT-5.2)DEFAME (GPT-4o)DEFAME (GPT-5.2)DEFAME Δ (pp)MMD-Agent (GPT-4V)MMD-Agent (GPT-5.2)MMD-Agent Δ (pp)
AVeriTeC80.870.569.4−1.1———
MMFakeBench84.2—71.4—61.664.7+3.1
MOCHEG87.159.264.3+5.1———
VERITE86.783.984.1+0.2———

This additional comparison provides a more direct assessment of RETRACE's verification effectiveness under a shared GPT-5.2 reasoning backbone and closely aligned experimental conditions. As shown in the table, RETRACE consistently outperforms DEFAME across all four benchmarks, achieving accuracy improvements of 11.4, 22.8, 2.6, and 12.8 percentage points on AVeriTeC, MOCHEG, VERITE, and MMFakeBench, respectively. The largest improvement is observed on MOCHEG, where RETRACE achieves 87.1% accuracy compared with DEFAME's 64.3%, suggesting that RETRACE's structured verification process may be particularly beneficial for evidence-intensive claim verification. On VERITE, the smaller improvement of 2.6 percentage points reflects the already strong performance of DEFAME (84.1%), while RETRACE further improves accuracy to 86.7%. Furthermore, on MMFakeBench, RETRACE achieves 84.2% accuracy, outperforming both DEFAME (71.4%) and MMD-Agent (64.7%) by 12.8 and 19.5 percentage points, respectively. These results provide additional evidence that RETRACE's improvements are not solely attributable to the underlying reasoning backbone, while we also acknowledge that differences in the verification procedures may also influence the observed performance.

To clearly distinguish reproduced results from previously published results, we have updated Table 2 by replacing the original DEFAME and MMD-Agent entries with their corresponding reproduced results, explicitly marked as “Reproduced,” while retaining citations to the original publications. The revised table caption also clarifies that these results were obtained using GPT-5.2 on the same benchmark samples as RETRACE.

Furthermore, our existing controlled ablation studies in Tables 4 and 5 of the revised manuscript provide complementary evidence regarding the contributions of RETRACE's two principal stages. These experiments examine alternative verification-strategy selection approaches and evidence-gap repair configurations under a common experimental setting, helping to isolate the contributions of the proposed mechanisms.

AuthorsOpenReview ↗

Response to Reviewer pCp7 (2)

Question 1: Were the prior methods in Table 2 rerun under a shared setup, or were their scores taken from the original papers? Are the evaluation samples and evidence conditions identical?

Thank you for raising this important question regarding the source of the baseline results and the consistency of the evaluation settings. We clarify that Table 2 includes both results obtained from the original published manuscripts and results reproduced under controlled experimental conditions. Specifically, we reproduced DEFAME using GPT-5.2 as the common reasoning backbone across all four benchmarks (AVeriTeC, MOCHEG, VERITE, and MMFakeBench) and MMD-Agent using the same backbone on MMFakeBench. These reproduced evaluations use the same benchmark samples and consistent label mappings as RETRACE. Importantly, the original DEFAME implementation evaluated MOCHEG on a different subset; our reproduction addresses this discrepancy by evaluating DEFAME on the same MOCHEG samples as RETRACE. The reproduced results are explicitly identified as “Reproduced” in Table 2, while the remaining baseline results are taken from their respective published manuscripts.

Regarding evidence conditions, we acknowledge that the originally reported methods may differ in their underlying models, retrieval tools, and external evidence sources. Although the evaluation samples are aligned, these differences mean that the evidence conditions cannot be considered strictly identical across all methods. We have therefore clarified in the revised manuscript which were obtained through our own reproduction to improve the transparency of Table 2 and allows readers to interpret the comparisons with a clearer understanding of their evaluation conditions.

AuthorsOpenReview ↗

Response to Reviewer pCp7 (3)

Questions 2: Can you provide comparisons against strong prior systems using the same backbone and comparable retrieval and inference budgets?

Thank you for this important question. We have addressed these concerns in detail in our responses to Critical Weaknesses 1 and 2, where we provide additional comparisons with strong prior systems using the same reasoning backbone and discuss the experimental settings. We kindly refer you to the particular response for the corresponding results and detailed explanations.

Reviewer_w2gAOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This is a work that offers a framework for fact-checking. Its novelty consists in having an extra step at the very beginning, asking "which strategy should we use?" (candidate strategies are ranked through pairwise LLM comparisons), and also combining the main strategy with an approach that questions the initial evidence, focusing on emotions-related information, that can skew the perception of a piece of information, and bias a reader. There are a couple of different systems that score the emotions of the input (a DistilRoBERTa text classifier and EmotionCLIP-V2 for images), but the overall assessment still relies on LLMs. The system is evaluated on AVeriTeC, MOCHEG, VERITE and MMFakeBench, with ablations of both stages.

Critical strengths

1° Intuitive and sensible research question: can we improve a system's ability in fact-checking by letting it actively reason on how to deal with the problem at hand, scoring possible approaches by the final plausible gain? The problem is clearly stated and the images and schemas helped in its understanding. One exception is Figure 3, which shows two labels per example without marking which is the gold label and which the prediction.

2° The paper mostly combines existing components rather than adding a new method, but the combination is well done, and each of the two stages is tested against alternatives. Stage I is compared against five other planning mechanisms under the same backbone and tools (Table 3), and the Stage II variants against a generic second retrieval with the same retrieval budget (Table 4). The final verifier sees only the claim and the evidence, so the strategy and emotion signals cannot act as shortcuts to the verdict. This care in the design is the main reason I lean towards acceptance.

3° The focus on emotions is well motivated, since many disinformation techniques work through emotional appeal and persuasion rather than through argument

Critical weaknesses

1° The system relies on model capabilities and therefore also on the model's biases. GPT-5.2 is called at almost every step: strategy generation and ranking, requirement extraction, coverage scoring, repair planning, joint emotion scoring and the final verdict. Given the high amount of passes done through stochastic systems to denote, choose, classify strategies, and pursue them, its reliability is hard to quantify. RETRACE is only run with GPT-5.2 (GPT-4o appears only as a direct baseline), so the statement in Appendix D that it can operate with other backbones is not tested. I would have liked less reliance on such closed LLMs, or at least tests with open-weight models that can run locally, so that the system does not depend on one company and on its model updates.

2° There is no assessment of stability across runs. No result is reported with variance or over repeated runs, while several differences in the ablations are small: coverage-guided and emotion-guided repair differ by 0.5–0.8 points across the four benchmarks (Table 4). Without repeated runs one cannot tell whether such differences exceed run-to-run variation. The authors list the stochastic components in limitation (ii) but do not measure their effect. Running the full pipeline several times on a subsample of each dataset would answer this. Stability across runs is undervalued in the field in general, and a fact-checking system presented as auditable should give the same verdict for the same claim.

3° There is no way to assess where a failure happened. The failure analysis lists five failure modes over 350 errors, and three of them point to a specific stage (strategy–task mismatch in Stage I, incomplete gap repair in Stage II, verification error in the final verifier). The paper gives no frequencies, so the reader cannot tell which stage fails most often, or which part of the pipeline to improve.

Questions for authors

1° How stable are the results overall? How often does the same input receive the same label across runs, and how often does Stage I select the same strategy? 2° How are the 350 analyzed errors distributed across the five failure modes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer w2gA (1)

Critical Strengths 1: One exception is Figure 3, which shows two labels per example without marking which is the gold label and which the prediction.

Thank you for pointing out this ambiguity in Figure 3. To address this, we have revised Figure 3 by clearly identifying the Gold Label and Predicted Label for each.

AuthorsOpenReview ↗

Response to Reviewer w2gA (2)

Critical Weaknesses 1: The system relies on model capabilities and therefore also on the model's biases. GPT-5.2 is called at almost every step: strategy generation and ranking, requirement extraction, coverage scoring, repair planning, joint emotion scoring and the final verdict. Given the high amount of passes done through stochastic systems to denote, choose, classify strategies, and pursue them, its reliability is hard to quantify. RETRACE is only run with GPT-5.2 (GPT-4o appears only as a direct baseline), so the statement in Appendix D that it can operate with other backbones is not tested. I would have liked less reliance on such closed LLMs, or at least tests with open-weight models that can run locally, so that the system does not depend on one company and on its model updates.

Thank you for highlighting the reliance on a proprietary backbone and the importance of evaluating RETRACE's generalizability across different vision-language models. To address this concern, we extended our evaluation to two open-weight vision-language models, Qwen3.8-27B and Llama-3.2-11B-Vision-Instruct, replacing GPT-5.2 as the reasoning backbone. To further clarify the experimental setup, we used Qwen3.8-27B and Llama-3.2-11B-Vision-Instruct as complete replacements for GPT-5.2 throughout the RETRACE framework. The results across all four benchmarks are summarized below.

DatasetGPT-5.2Qwen3.8-27BLlama-3.2-11B-Vision-InstructStrongest Baseline (Table 2)
AVeriTeC80.878.471.278.0 (CAAFC)
MOCHEG87.184.177.264.3 (DEFAME)
VERITE86.782.374.184.1 (DEFAME)
MMFakeBench84.282.176.375.3 (T²Agent)

With Qwen3.8-27B, accuracy drops by only 2.1–4.4 points and RETRACE still beats the strongest baselines on AVeriTeC (+0.4), MOCHEG (+19.8) and MMFakeBench (+6.8). The smaller Llama-11B model drops further (7.9–12.6 points), suggesting that performance scales with backbone capacity. Together with the Qwen results, this indicates that RETRACE’s gains are not specific to GPT-5.2.

AuthorsOpenReview ↗

Response to Reviewer w2gA (3)

Critical Weakness 2: There is no assessment of stability across runs. No result is reported with variance or over repeated runs, while several differences in the ablations are small: coverage-guided and emotion-guided repair differ by 0.5–0.8 points across the four benchmarks (Table 4). Without repeated runs one cannot tell whether such differences exceed run-to-run variation. The authors list the stochastic components in limitation (ii) but do not measure their effect. Running the full pipeline several times on a subsample of each dataset would answer this. Stability across runs is undervalued in the field in general, and a fact-checking system presented as auditable should give the same verdict for the same claim.

Question 1: How stable are the results overall? How often does the same input receive the same label across runs, and how often does Stage I select the same strategy?

We would like to thank the reviewer for highlighting the importance of evaluating run-to-run stability, particularly for a multi-stage verification framework involving stochastic model decisions. To address this concern, we extend our analysis along three dimensions: (1) overall accuracy stability across repeated executions, (2) consistency of final verification labels, and (3) consistency of Stage I verification-strategy selection.

  1. Overall Performance Stability Across Runs

To assess the reliability of RETRACE's overall verification performance, we conducted three independent evaluation runs on the full evaluation set of each of the four benchmarks, rather than using a smaller evaluation subset. All runs followed the same experimental configuration, including the reasoning backbone, prompts, decoding settings, verification pipeline, and retrieval configuration. Run 1 corresponds to the original results reported in Table 2 of the submitted manuscript. These results have also been incorporated into the revised manuscript to explicitly quantify run-to-run variability. The results are presented below.

RunAVeriTeCMOCHEGVERITEMMFakeBench
Run 180.887.186.784.2
Run 281.487.287.084.0
Run 380.986.886.984.5
Mean ± Std.81.03 ± 0.3287.03 ± 0.2186.87 ± 0.1584.23 ± 0.25

Standard deviations are ≤ 0.32 pp, and the largest difference between runs is 0.3–0.6 pp.

Regarding the small evidence-gap repair ablation differences in Table 5 of the revised manuscript, we acknowledge that the 0.5–0.8-percentage-point differences between coverage-guided and emotion-guided repair are comparable in magnitude to the observed run-to-run accuracy ranges of 0.3–0.6 percentage points. Importantly, the main contribution of RETRACE lies in combining these complementary repair signals rather than establishing the superiority of one over the other. The combined Full-EGR configuration consistently outperforms coverage-guided repair by 1.3–2.1 percentage points and emotion-guided repair by 1.8–2.8 percentage points across the four benchmarks. Notably, on AVeriTeC, MOCHEG, and VERITE, these improvements exceed the maximum observed run-to-run accuracy variations of 0.6, 0.4, and 0.3 percentage points, respectively. This provides stronger empirical evidence that the improvement achieved by combining both signals is larger than the observed stochastic variation in overall verification performance.

  1. Consistency of Final Verification Predictions

To investigate prediction-level repeatability, we randomly sampled 100 claims from each of the four benchmarks, resulting in 400 claims evaluated across three independent runs under the same experimental configuration. As illustrated in the below table, 349 claims (87.25%) receive identical labels across all three runs, while 50 claims (12.50%) receive the same label in two runs and a different label in the remaining run. Only one claim (0.25%) receives three different labels. A three-run majority vote therefore gives a definite label for 99.75% of claims, and claims where runs disagree can be flagged for human review. This supports auditable use of RETRACE.

Prediction ConsistencySamplesPercentage
All three runs agree34987.25%
Two runs agree; one differs5012.50%
All three runs disagree10.25%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.