acceptodds
Under review as a conference paper at ICLR 2027

CAUTION: Calibrated Uncertainty-Guided Sample-Adaptive Evidence Reasoning for Short-Video Fake News Detection

Abstract

Short-video fake news samples vary substantially in verification difficulty: some can be reliably assessed from video-internal multimodal cues, whereas factually false yet internally coherent cases require external evidence. Existing methods nevertheless assign a fixed reasoning depth to all samples, either under-verifying evidence-dependent claims or unnecessarily subjecting otherwise reliable predictions to costly and potentially harmful slow deliberation. To address these challenges, we propose CAUTION, a calibrated uncertainty-guided framework for adaptive-depth evidence reasoning. First, a lightweight multimodal fast system exploits video-internal textual, visual, and acoustic cues to form an initial veracity estimate. Second, a calibrated adaptive-depth controller uses label-conditional conformal prediction to route uncertain samples to slow deliberation and trajectory-level calibration to determine when deliberation should stop. Third, a progressive relational evidence-revision module incrementally organizes ranked claim–evidence analysis units into a sample-specific heterogeneous graph, where typed support, refute, and unresolved relations capture their argumentative roles. Relation-aware graph attention aggregates the evolving evidence, while a reliability-gated residual revises rather than replaces the initial estimate. We further construct SV-CoRE, a large-scale claim-oriented evidence repository aligned with FakeSV and FakeTT, through retrieval, cleaning, deduplication, ranking, and large-model claim–evidence analysis. Experiments on both benchmarks demonstrate that CAUTION achieves state-of-the-art detection performance while substantially reducing unnecessary slow deliberation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fake News Detection, Multimodal Learning, Conformal Prediction, Selective Prediction, Evidence-Grounded Reasoning, Adaptive Computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Tzx1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces CAUTION, a method for detecting fake news in short videos. It first makes an initial prediction using the video’s text, images, and audio. A conformal prediction rule then decides which videos need further checking. For these cases, the system examines external evidence for individual claims, adds the findings to a graph, and updates its prediction. A learned gate controls how much the evidence changes the initial decision, while a stopping rule decides whether more evidence is needed. The authors also build SV-CoRE, a repository of claims and evidence analyses. Experiments on FakeSV and FakeTT compare accuracy, evidence usage, and different components of the method. The reported results show that selective checking can improve accuracy while using fewer evidence analyses than processing all available evidence.

Critical strengths
  1. I find the focus on deciding when additional evidence is useful particularly worthwhile. The method addresses both the cost of further checking and the risk of changing an initially correct prediction. Combining selective evidence gathering with controlled updates gives the approach a clear practical purpose: spend more effort on difficult cases while limiting unnecessary revisions.

  2. The experiments examine how often evidence corrects a mistake or harms an initially correct prediction, alongside accuracy and evidence usage. These measurements help explain why selective checking can be useful. Comparisons with fast-only prediction, full-evidence processing, and confidence-based routing, together with the component ablations, provide useful support for the design. The fixed SV-CoRE evidence repository also provides a basis for repeatable comparisons.

Critical weaknesses

The main idea is understandable, but I found the method section difficult to follow, especially how the initial prediction, evidence gathering, and later revisions fit together. Several components are introduced with little explanation of their purpose before the equations. For example, TfastT_{\mathrm{fast}} appears in Equation (3) before its role is explained, while FLERF_{\mathrm{LER}} and a(r)a^{(r)} require more explicit descriptions.

I also struggled with the conformal routing and reliability-gated revision sections. The mathematical descriptions are provided, but the intuition behind the thresholds, counterfactual graph, and correction gate is harder to follow. Moving between the main text and appendices makes it difficult to keep track of the complete process. A short example would help explain how these components work together and why each is needed.

The metric reporting in Table 2 needs clarification. For Qwen-VL on FakeTT, the reported accuracy is 58.86%, while macro-F1 is 73.89%. Under the standard binary definition, macro-F1 cannot exceed accuracy when both are computed from the same predictions. This could be a labeling issue or a difference in how the metrics were calculated, but it raises a concern about whether the F1 scores are directly comparable across methods. The authors should clarify the averaging convention and verify that it is applied consistently across the reported results.

The controller is a key part of the method, but I could not find results showing how often its set of possible labels includes the correct answer, separately for real and fake videos. It would help to report this and show how changing the allowed error level affects accuracy and the amount of evidence used. These checks are especially useful because the rule is calibrated using older videos and tested on newer ones. The theoretical guarantee assumes these groups follow similar statistical patterns, which may become less reliable as video content changes over time.

Questions for authors
  1. I was confused by the Qwen-VL results on FakeTT: accuracy is 58.86%, while macro-F1 is 73.89%. Could you explain how these were calculated? Is the F1 score for one class or an average over both, and is the same definition used for all methods?

  2. Could you show how often the conformal prediction sets include the correct label for real and fake videos separately? I would like to see how these rates compare with the intended coverage on the chronological test splits.

  3. Could you walk through one example to explain how the initial prediction, evidence gathering, revision, and stopping steps fit together, and why each component is needed? Clearer explanations of all terms, e.g. TfastT_{\mathrm{fast}}, FLERF_{\mathrm{LER}}, a(r)a^{(r)} when they first appear would also make the method easier to follow.

-My main reservation is that the controller’s calibration is a central contribution, yet its empirical coverage is not reported. Together with the unresolved metric definitions, this leaves me insufficiently convinced by the current evaluation. Clarifying these would be helpful for me.

10 comments
AuthorsOpenReview ↗

Weakness 3 & Question 1: Qwen-VL F1 Correction

Thank you for identifying this reporting error. The intended metric in Table 2 is Macro-F1, the unweighted mean of the Fake-class and Real-class F1 scores. The Qwen-VL row mistakenly listed Fake-class F1 on FakeSV (0.7635) and Real-class F1 on FakeTT (0.7389). The original Precision and Recall were also class-specific. The corrected row uses two-class macro averages for F1, Precision, and Recall, based on the reported metrics for the complete original test sets (FakeSV: 542 videos; FakeTT: 299 videos).

\qquad F_{1,c}=\frac{2TP_c}{2TP_c+FP_c+FN_c}.$$ $$\begin{array}{lrrrrrrrr} \hline & & \rlap{\kern-1em\textbf{FakeSV}} & & & & \rlap{\kern-1em\textbf{FakeTT}} & & \\\\ \textbf{Model} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Precision} & \textbf{Recall} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Precision} & \textbf{Recall} \\\\ \hline \text{Qwen-VL} & 0.7417 & 0.7395 & 0.7388 & 0.7415 & 0.5886 & 0.3852 & 0.3567 & 0.4451 \\\\ \hline \end{array}$$ While checking Table 2 of the original manuscript, we also found that the FakingRecipe entry on FakeSV reported weighted F1 (0.8192) instead of macro-F1 (0.8144). We will correct the Qwen-VL row and the FakeSV FakingRecipe entry in the revised Table 2. Thank you again for pointing this out.
AuthorsOpenReview ↗

Weakness 4 & Question 2 (Part 1/3): Chronological Splits

Thank you for pointing out that changes in video content between the earlier calibration set and the later test set could affect how well the rule works. Your comment prompted us to measure how often the prediction sets included the correct label on the later test splits. Below, we document the publication-date ranges for the chronological splits (Table R3.1). We then compare coverage for real and fake videos with the intended level, both before using any evidence and across all evidence rounds (Table R3.2). We also show how changing the allowed error level affected accuracy and how much evidence the system used (Table R3.3).


All models are trained, calibrated, and tested strictly in chronological order. Auditing the video publication timestamps (UTC+08:00) across the official splits confirms zero temporal overlap between calibration (validation) and test periods:

Table R3.1: Chronological publication-date horizons across official benchmark splits.

\hline \textbf{Benchmark (Train / Val / Test }N\textbf{)} & \textbf{Training} & \textbf{Validation / Calibration} & \textbf{Held-out test} \\\\ \hline \textbf{FakeSV}\text{ (2,536 / 546 / 542)} & \text{2017-10-08 – 2021-06-26} & \text{2021-06-26 – 2021-10-09} & \textbf{2021-10-09 – 2022-02-06} \\\\ \textbf{FakeTT}\text{ (1,393 / 299 / 299)} & \text{2019-05-06 – 2023-10-20} & \text{2023-10-20 – 2024-01-18} & \textbf{2024-01-18 – 2024-03-13} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Weakness 4 & Question 2 (Part 2/3): Class-Wise and Overall Empirical Coverage Across α

We evaluate empirical coverage (the fraction of test videos whose ground-truth label YY is included in the calibrated prediction set) at two levels:

  • Round 0 (r=0r=0; N=542N=542 / 299299): Whether Y∈Γ(0)(X)Y \in \Gamma^{(0)}(X) before any external evidence is retrieved (evaluated on 100% of test videos).
  • Evidence Rounds (r≥1r \ge 1; N=537N=537 / 288288): Each added claim--evidence unit produces a revised prediction and prediction set. A video is covered only if its true label belongs to every prediction set across all available evidence rounds, including rounds beyond the actual stopping point. The 5 FakeSV and 11 FakeTT videos without an evidence prefix are excluded only from this coverage calculation and remain in all classification metrics.

Table R3.2: Empirical test coverage across allowed error levels α\alpha on chronological test splits.

\hline & & & \textbf{Round 0 }(r=0) & & \hspace{0.3em} & & \textbf{Evidence Rounds }(r \ge 1) & \\\\ \textbf{Dataset} & \alpha & \textbf{Fake} & \textbf{Real} & \textbf{Overall} & & \textbf{Fake} & \textbf{Real} & \textbf{Overall} \\\\ \hline \textbf{FakeSV} & 0.03 & 99.67\\% & 94.96\\% & 97.60\\% & & 99.67\\% & 94.87\\% & 97.58\\% \\\\ & \mathbf{0.05} & \mathbf{99.01\\%} & \mathbf{93.28\\%} & \mathbf{96.49\\%} & & \mathbf{99.34\\%} & \mathbf{94.44\\%} & \mathbf{97.21\\%} \\\\ & 0.07 & 99.01\\% & 86.97\\% & 93.73\\% & & 99.01\\% & 91.45\\% & 95.72\\% \\\\ & 0.10 & 97.70\\% & 80.25\\% & 90.04\\% & & 97.69\\% & 88.03\\% & 93.48\\% \\\\[3pt] \textbf{FakeTT} & 0.03 & 95.96\\% & 100.0\\% & 98.66\\% & & 95.74\\% & 100.0\\% & 98.61\\% \\\\ & \mathbf{0.05} & \mathbf{88.89\\%} & \mathbf{100.0\\%} & \mathbf{96.32\\%} & & \mathbf{90.43\\%} & \mathbf{100.0\\%} & \mathbf{96.88\\%} \\\\ & 0.07 & 86.87\\% & 96.50\\% & 93.31\\% & & 86.17\\% & 96.91\\% & 93.40\\% \\\\ & 0.10 & 83.84\\% & 91.00\\% & 88.63\\% & & 86.17\\% & 90.21\\% & 88.89\\% \\\\ \hline \end{array}$$ * **Evaluating the reviewer's temporal-shift concern:** At our default $\alpha=0.05$, while aggregate coverage exceeds 95% (`96.32%`–`97.21%`), **class-wise empirical coverage is asymmetric**: `FakeSV-Fake` (`99.01%` Round 0) and `FakeTT-Real` (`100.0%` at Round 0 and across all evidence rounds) exceed 95%, whereas `FakeSV-Real` (`93.28%` at Round 0; `94.44%` across evidence rounds) and `FakeTT-Fake` (`88.89%` at Round 0; `90.43%` across evidence rounds) fall below the 95% nominal level. To examine this potential temporal shift, we compare the nonconformity-score distributions between calibration and test splits. On FakeTT-Fake at $\alpha=0.05$, the fraction of Round-0 scores exceeding $\widehat q_{\text{Fake}}^{(0)}$ increases from 4.94% (4/81) on the validation calibration split to 11.11% (11/99) on the later test split. This difference is consistent with a possible temporal distribution shift, alongside finite-sample variability. * **Effect of adjusting $\alpha$:** Lowering $\alpha$ increases empirical coverage by making the conformal prediction sets more inclusive, without retraining either prediction model. In particular, reducing $\alpha$ from $0.10$ to $0.03$ raises Round-0 coverage by 14.71 pp on FakeSV-Real (80.25% → 94.96%) and 12.12 pp on FakeTT-Fake (83.84% → 95.96%).
Reviewer_3JxNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a CAUTION framework to solve the problem that short video false news verification is difficult and unified reasoning depth is easy to cause insufficient verification or computational waste. This method uses a lightweight multi-mode fast system to form an initial prediction, decides whether to start external evidence verification by label condition conformal prediction, and adaptively stops reasoning in combination with trajectory calibration. The slow system gradually builds the statement evidence heterogeneous graph, and corrects the initial judgment through relationship perception attention and reliability gated residual. The author also constructed an SV CoRE resource containing 27699 evidence analysis units, and verified the detection effect and evidence use efficiency on FakeSV and FakeTT.

Critical strengths
  1. The paper distinguishes between samples that can be judged by internal clues of videos and samples that need external fact verification, and avoids performing unified verification on all videos through selective reasoning. The experiment further counts the number of error correction and error correction, which helps to explain why adaptive verification is better than undifferentiated use of evidence.

  2. Methods Gradually organize statements and evidence analysis, distinguish support, refutation and unresolved relationships, and control the correction extent of evidence to initial prediction through gated residuals. This design not only uses external information, but also considers the interference to correct judgment when the evidence is insufficient or unreliable.

  3. SV CoRE keeps the statement, retrieval source and structural analysis to facilitate the recurrence of progressive evidential reasoning. The paper also compares fast systems, complete evidence, fixed depth and different routing strategies, showing the trade-off between performance and evidence usage.

Critical weaknesses

1.The statistical assurance of the complete control process and the actual role of the stop module are not clear. Conformal coverage constraints are not directly equivalent to the conditional error rate of exiting samples, and the final results are also affected by trajectory maintenance, relationship consistency check, and budget exhaustion rollback. It is necessary to report the independent trigger rate of each stop condition, and define the event and probability bounds actually guaranteed by the complete system.

2.Some performance improvements are small, and there is still insufficient statistical evidence. The accuracy rate on FakeTT is only 0.67 percentage points higher than that on the fast system, corresponding to a net correction of about two samples; Compared with the best comparison method, the increase is only 0.33 percentage points. Although the average value of three random seeds is taken as the result, the standard deviation, confidence interval or paired significance test is not reported, so it is difficult to judge whether the advantage is stable.

3.Not entering a tag into the searcher does not rule out that the search result contains a rumor refuting conclusion directly related to the original video or information after the event. The paper has not yet fully analyzed the overlap of evidence time and sources. Some baselines did not obtain the same external evidence, and CSI adopted local adaptation.

Questions for authors

1.On FakeSV and FakeTT, what are the actual calibration samples, final threshold and independent trigger proportion of the track keeping controller? How much is the gain compared to using conformal singleton stop only?

2.What are the proportions of fast exit, slow single instance exit, track keeping exit, and budget exhausted rollback? What is the empirical error rate of each branch and the final system?

3.Is the performance improvement on FakeTT significant under different random seeds and paired tests? After strictly matching the trigger rate or average evidence usage, is conformal routing still better than the category conditional confidence threshold?

6 comments
AuthorsOpenReview ↗

Weakness 1 & Question 1 (Part 1/2) : How Is Preservation Stopping Calibrated?

Thank you for recognizing the value of selective verification and for asking what the preservation rule adds beyond conformal stopping. Compared with conformal-only stopping, adding the preservation rule reduces average evidence use per test video by 16.03% on FakeSV and 42.25% on FakeTT without changing any final predictions (Table R2.2).

(1) How Is Preservation Stopping Calibrated?

In the slow system, conformal stopping requires a prediction set containing only the current predicted label. Preservation stopping instead estimates whether stopping now would yield the same final prediction as the conformal-only policy, so it can accept even without a singleton set. In a new controller audit (Table R2.1), we documented the preservation calibration settings and measured each rule's acceptance on routed test videos.

Table R2.1. Preservation calibration and acceptance by stopping rule.

\begin{array}{lcccccc} \hline \textbf{Test split} & \begin{array}{c}\textbf{Eligible}\\\\\textbf{val. trajectories}\end{array} & \begin{array}{c}\textbf{Calibrated}\\\\\widehat q_{\mathrm{stop}}\end{array} & \begin{array}{c}\textbf{Final}\\\\\tau_{\mathrm{stop}}\end{array} & \begin{array}{c}\textbf{Earliest}\\\\\textbf{pres. round}\end{array} & \begin{array}{c}\textbf{Conformal}\\\\\textbf{accepts/routed}\end{array} & \begin{array}{c}\textbf{Preservation}\\\\\textbf{accepts/routed}\end{array} \\\\ \hline \begin{array}{l}\text{FakeSV}\\\\(N=542)\end{array} & 228 & 0.979448 & 0.990000 & 1 & 40/264\\,(15.15\\%) & 111/264\\,(42.05\\%) \\\\ \begin{array}{l}\text{FakeTT}\\\\(N=299)\end{array} & 122 & 0.999829 & 0.999829 & 2 & 0/137\\,(0.00\\%) & 75/137\\,(54.74\\%) \\\\ \hline \end{array}$$ Calibration uses routed validation videos with at least one evidence round. The final threshold $\tau_{\mathrm{stop}}$ is the larger of the calibrated quantile $\widehat q_{\mathrm{stop}}$ and the preset minimum of 0.99. *Earliest pres. round* is the first round at which the preservation rule is allowed to stop inference. For each rule separately, *accepts/routed* counts the share of routed test videos that satisfy the rule at least once, including at the final evidence round. One FakeSV video satisfies both rules, with conformal stopping first; Table R2.3 reports the actual exits. On FakeTT, **conformal prediction governs Round-0 routing**, letting **162/299 videos (54.18%)** exit without evidence. The **0/137** applies only to slow-stage conformal singleton stopping; **75/137** routed videos meet the preservation criterion.
AuthorsOpenReview ↗

Weakness 1 & Question 1 (Part 2/2) : What Does It Add Beyond Conformal Stopping?

(2) What Does It Add Beyond Conformal Stopping?

A rule can accept at the last round without saving evidence. To measure the benefit of adding preservation stopping, Table R2.2 extends the paper's FakeSV stopping ablation to FakeTT, holding entry routing, checkpoints, evidence order, gated revision, and relation checking fixed.

Table R2.2. Slow stopping ablation (average evidence units per test video).

\begin{array}{llrrrr} \hline \textbf{Test split} & \textbf{Slow stopping} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Avg. units} & \textbf{Correct / Hurt} \\\\ \hline \text{FakeSV }(N=542) & \text{Conformal only} & 0.8708 & 0.8670 & 3.074 & 49/12 \\\\ & \text{Preservation only} & 0.8745 & 0.8708 & 3.020 & 50/11 \\\\ & \textbf{Both (CAUTION)} & 0.8708 & 0.8670 & {\color{maroon}\mathbf{2.581}} & 49/12 \\\\[0.4em] \text{FakeTT }(N=299) & \text{Conformal only} & 0.8428 & 0.8264 & 1.813 & 3/1 \\\\ & \text{Preservation only} & 0.8428 & 0.8264 & \mathbf{1.047} & 3/1 \\\\ & \textbf{Both (CAUTION)} & 0.8428 & 0.8264 & {\color{maroon}\mathbf{1.047}} & 3/1 \\\\ \hline \end{array}$$ **Stopping-rule contributions (Table R2.2).** On FakeSV, the combined rules use **14.5% fewer evidence units** than `Preservation only` (3.020 to 2.581 per video), with two fewer correct predictions. On FakeTT, preservation accounts for all **63 stops before evidence exhaustion**, cutting usage from **1.813 to 1.047 units per test video** with no change in final predictions versus conformal-only stopping. **Full-system outcome on FakeTT (paper Table 3).** Conformal routing decides whether to seek evidence, stopping controls how much to gather, and gated revision and relation checks govern prediction changes. Using all evidence yields **83.28%** accuracy, below `Fast-only` (**83.61%**). CAUTION reaches **84.28%** using **77.24% fewer evidence units** than `Always all evidence`. Both evidence policies correct three fast-model errors, but CAUTION makes **one rather than four harmful revisions**. **(3) Which Errors Does Calibration Actually Bound?** Neither calibration level guarantees the overall classification error rate or the error rate within an exit branch. When the scoring rules are fixed before calibration and calibration and test data are exchangeable within each class, the Round-0 singleton rule gives the following bound at $\alpha=0.05$: $$\Pr(\widehat Y_{\mathrm{fast}}\ne Y,\\ \mathrm{fast\\ exit}\mid Y=\ell) \le \Pr(Y\notin\Gamma^{(0)}(X)\mid Y=\ell)\le \alpha.$$ Under the same assumptions, trajectory calibration also bounds the probability of excluding the true label at **any** monitored slow round by $\alpha$. The preservation level $\alpha_{\mathrm{stop}}=0.01$ targets a different event: a preservation stop returns a relation-checked label that **disagrees with the conformal-only reference decision**.
AuthorsOpenReview ↗

Weakness 1 & Question 2: How Often Does Each Exit Occur, and What Are the Branch and Overall Error Rates?

We appreciate your request to break down the exit outcomes, since the overall error rate alone can hide differences between branches.

Our new exit analysis (Table R2.3) separates Round-0 routing from slow-system exits and reports branch and overall error rates.

Table R2.3. First exits and errors on the same videos.

\begin{array}{lrrrr} \hline \textbf{Exit branch} & \begin{array}{c}\textbf{FakeSV }(N=542)\\\\\textbf{Count (share)}\end{array} & \begin{array}{c}\textbf{FakeSV errors}\\\\\text{Fast}\to\text{CAUTION}\end{array} & \begin{array}{c}\textbf{FakeTT }(N=299)\\\\\textbf{Count (share)}\end{array} & \begin{array}{c}\textbf{FakeTT errors}\\\\\text{Fast}\to\text{CAUTION}\end{array} \\\\ \hline \text{Fast exit} & 278\\;(51.29\\%) & 19\to19\\;(6.83\\%) & 162\\;(54.18\\%) & 11\to11\\;(6.79\\%) \\\\ \text{Conformal stop} & 40\\;(7.38\\%) & 30\to13\\;(32.50\\%) & 0\\;(0.00\\%) & \text{—} \\\\ \text{Preservation stop} & 110\\;(20.30\\%) & 6\to6\\;(5.45\\%) & 75\\;(25.08\\%) & 26\to24\\;(32.00\\%) \\\\ \begin{array}{l}\text{Exhausted /}\\\\\text{no evidence}\end{array} & 114\\;(21.03\\%) & 52\to32\\;(28.07\\%) & 62\\;(20.74\\%) & 12\to12\\;(19.35\\%) \\\\[0.4em] \textbf{All routed} & \mathbf{264\\;(48.71\\%)} & \mathbf{88\to51\\;(19.32\\%)} & \mathbf{137\\;(45.82\\%)} & \mathbf{38\to36\\;(26.28\\%)} \\\\ \textbf{Complete system} & \mathbf{542\\;(100\\%)} & \mathbf{107\to70\\;(12.92\\%)} & \mathbf{299\\;(100\\%)} & \mathbf{49\to47\\;(15.72\\%)} \\\\ \hline \end{array}$$ *Exit shares use the full test split. The error columns compare Fast-only with CAUTION on the same videos and give CAUTION's error rate within each branch.* On FakeTT, conformal routing lets **162/299 videos exit at Round 0 without evidence**. The zero applies only to slow conformal stopping; preservation accounts for all **63 stops before evidence exhaustion**. Together, routing and stopping adapt whether to seek evidence and when to stop for each video. **When evidence runs out.** If neither stopping rule fires, CAUTION uses the last evidence round's prediction. If it differs from the fast prediction, CAUTION keeps the change if it passes the relation check and returns the fast prediction otherwise. With no evidence, CAUTION uses the fast prediction directly. Of 111 exhausted FakeSV videos, **34 kept a changed label** and **20 reverted to the fast prediction**. None of the **57** exhausted FakeTT videos reverted.
Reviewer_iRhLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAUTION, a sample-adaptive framework for short-video fake news detection. It uses a lightweight multimodal fast system for initial predictions and routes only uncertain samples to a slow system via label-conditional conformal prediction. The slow system iteratively retrieves claim-level evidence, builds a heterogeneous graph, and revises predictions via a gated residual mechanism. The authors also introduce SV-CoRE, a large-scale evidence repository. Experiments show CAUTION achieves state-of-the-art performance on FakeSV and FakeTT while routing fewer than 50% of samples to slow deliberation, significantly reducing computational overhead.

Critical strengths
  1. Statistically grounded adaptive routing: The use of label-conditional conformal prediction to govern the fast-to-slow routing is a strong design choice. Unlike heuristic confidence thresholds, this provides distribution-free statistical guarantees for fast-exit errors. This is consequential because it makes the sample-adaptive depth allocation theoretically sound and reliable, rather than purely empirical.
  2. Robust evidence integration via gated revision The reliability-gated residual revision effectively addresses the noise inherent in web retrieval. By scaling the evidence-induced correction via a learned gate rather than overwriting the initial prediction, the model successfully limits harmful revisions. This is well-supported by the ablation results, proving the framework's robustness against misleading external evidence.
Critical weaknesses
  1. Vulnerability to cascading upstream errors: The slow deliberation pipeline heavily relies on upstream LVLMs for claim extraction and LLMs for evidence analysis. If the LVLM extracts an irrelevant claim, or the LLM hallucinates a "support" relation for contradictory evidence, this error is structurally embedded into the graph. The paper lacks any internal consistency check or mitigation mechanism for such cascading failures, which poses a significant risk to the framework's reliability in real-world scenarios.
  2. Missing end-to-end latency analysis: While the paper convincingly shows a reduction in the number of LLM calls (Avg. Units), it completely lacks quantitative analysis of actual end-to-end inference latency (e.g., wall-clock time or FLOPs). Given the multi-stage pipeline (multimodal encoding, LVLM extraction, iterative GNN updates, and conformal checks), it is unclear whether the framework is genuinely efficient in a real-time setting, weakening the core claim of computational efficiency.
Questions for authors
  1. How does the system handle cases where the upstream LVLM extracts an irrelevant claim or the LLM misclassifies the evidence relation (e.g., hallucinating "support" for a refutation)? Are there any internal consistency checks to prevent this from corrupting the graph and the final prediction?
  2. Could you provide the average end-to-end inference latency (in seconds or milliseconds) per video for CAUTION compared to the "Fast only" and "Always all evidence" baselines? The current "Avg. Units" metric does not reflect actual computational time or system overhead, which is critical for evaluating the claimed efficiency.
3 comments
AuthorsOpenReview ↗

Weakness 1 & Question 1: How CAUTION Handles Upstream LVLM/LLM Errors

Thank you for recognizing the strengths of our work. We agree that upstream LVLMs and LLMs can occasionally extract peripheral claims or misclassify noisy web snippets. Therefore, CAUTION is explicitly designed to treat upstream VLM/LLM outputs as noisy proposals rather than ground truth, filtering them through three safeguards before they can alter the final prediction:

Prompting with factual anchors and explicit abstention:

Claim extraction prompts target checkable factual propositions (anchored in subject, time, and location) rather than subjective summaries. During evidence analysis, FLERF_{\mathrm{LER}} is instructed to output unresolved when retrieved snippets are mismatched or insufficient, rather than forcing a support/refute judgment. This allows uncertain evidence analyses to be explicitly represented in the graph.

Reliability gating and looping past unhelpful claims:

Even when a peripheral claim or noisy analysis a(r)\mathbf{a}^{(r)} enters the graph, our lightweight graph model cross-attends a(r)\mathbf{a}^{(r)} with the video's (T,V,A)(T,V,A) features and computes a counterfactual difference:

Δhg(r)=hg(r)−hg,cf.\Delta\mathbf{h}_g^{(r)}=\mathbf{h}_g^{(r)}-\mathbf{h}_{g,\mathrm{cf}}.

A supervised gate g(r)g^{(r)} is trained to suppress revisions that would degrade the fast predictor's performance. These mechanisms are designed to limit the influence of unhelpful claims, making the conformal set more likely to remain non-singleton (Γ(r)(x)=Fake,Real\Gamma^{(r)}(x)=\\{\texttt{Fake},\texttt{Real}\\}). In such cases, the controller continues to the next prioritized claim c(r+1)c^{(r+1)} rather than committing to an uncertain revision.

Moreover, each round computes its correction relative to the original fast scores, rather than accumulating corrections from Round r−1r-1, reducing the risk of compounding score-level errors across rounds:

z(r)=zfast+g(r)Δzevi(r).\mathbf{z}^{(r)}=\mathbf{z}_{\mathrm{fast}}+g^{(r)}\Delta\mathbf{z}_{\mathrm{evi}}^{(r)}.

Hard consistency check before label flipping:

Directly addressing your question about internal consistency checks, CAUTION enforces a hard constraint on label revisions (Sec. 2.3; Appx. B.4): CAUTION accepts a flip to Real only if support relations are at least as numerous as refute relations, and a flip to Fake only if refute relations are at least as numerous as support relations. Otherwise, it retains the fast prediction. This prevents score-level revisions from flipping the prediction against a majority of opposing evidence relations, without assuming that every upstream relation is correct. It thus guards against label flips driven by an isolated relation hallucination or an erratic logit shift.

Empirical evidence: The ablations in Table R1.1 show the value of controlling evidence-based revisions: on FakeSV, removing relation-aware edges reduces Net gains from +37 to +32 and removing gated revision increases harmful flips (Hurt) from 12 to 15; on FakeTT, removing Lpres\mathcal{L}_{\mathrm{pres}} drops Net from +2 to -3. We will clarify these safeguards in the revised paper.

Table R1.1. Selected evidence-revision ablations (main paper, Table 4).

\hline \textbf{Original test split} & \textbf{Variant} & \textbf{Acc. (}\\%\textbf{)} & \textbf{Correct} & \textbf{Hurt} & \textbf{Net} \\\\ \hline \text{FakeSV (542 videos)} & \textbf{Full CAUTION} & \mathbf{87.08} & 49 & 12 & \mathbf{+37} \\\\ & \text{Without relation-aware edges} & 86.35 & 43 & 11 & +32 \\\\ & \text{Without gated residual revision} & 86.72 & 50 & {\color{maroon}\mathbf{15}} & +35 \\\\ & \text{Without preservation loss} & 86.35 & 46 & 13 & +33 \\\\[4pt] \text{FakeTT (299 videos)} & \textbf{Full CAUTION} & \mathbf{84.28} & 3 & 1 & \mathbf{+2} \\\\ & \text{Without relation-aware edges} & 83.61 & 1 & 0 & +1 \\\\ & \text{Without gated residual revision} & 84.28 & 3 & 1 & +2 \\\\ & \text{Without preservation loss} & 82.61 & 1 & 4 & {\color{maroon}\mathbf{-3}} \\\\ \hline \end{array}$$ *Correct counts fast-system errors fixed by evidence; Hurt counts initially correct predictions changed to errors; Net = Correct − Hurt.*
AuthorsOpenReview ↗

Weakness 2 & Question 2: Inference Latency per Video

Thank you for your thoughtful review. We compared all three policies on the same 50 videos per dataset from the original test splits. As shown in Table R1.2, CAUTION's mean latency was 61.4% lower on FakeSV and 67.0% lower on FakeTT than Always all evidence.

Table R1.2. End-to-End Latency (seconds per video)

\hline \textbf{Test subset} & \textbf{Fast only} & \textbf{CAUTION} & \textbf{Always all evidence} \\\\ \hline \text{FakeSV (50 of 542)} & 1.77 & {\color{maroon}\mathbf{7.46}} & 19.34 \\\\ \text{FakeTT (50 of 299)} & 3.87 & {\color{maroon}\mathbf{9.48}} & 28.70 \\\\ \hline \end{array}$$ **Table R1.3. Latency Breakdown (seconds per video)** $$\begin{array}{llrrrrr} \hline \textbf{Dataset} & \textbf{Policy} & \begin{array}{c}\textbf{Feature}\\\\\textbf{extraction}\end{array} & \begin{array}{c}\textbf{Claim}\\\\\textbf{extraction}\end{array} & \textbf{Retrieval} & \begin{array}{c}\textbf{Qwen}\\\\\textbf{analysis}\end{array} & \begin{array}{c}\textbf{Graph}\\\\\textbf{inference}\end{array} \\\\ \hline \text{FakeSV} & \text{Fast only} & 1.7627 & 0 & 0 & 0 & 0.0061 \\\\ & \textbf{CAUTION} & 1.7627 & 1.8019 & 0.0670 & 3.7744 & 0.0519 \\\\ & \text{Always all evidence} & 1.7627 & 5.7422 & 0.2504 & 11.4728 & 0.1117 \\\\[4pt] \text{FakeTT} & \text{Fast only} & 3.8597 & 0 & 0 & 0 & 0.0056 \\\\ & \textbf{CAUTION} & 3.8597 & 2.0557 & 1.1307 & 2.3952 & 0.0370 \\\\ & \text{Always all evidence} & 3.8597 & 5.9496 & 2.7329 & 16.0393 & 0.1182 \\\\ \hline \end{array}$$ **Timing setup.** We used the same frozen checkpoint per dataset (RTX 3090). Qwen3.8-Flash extracted claims and Qwen3.6-Plus produced short evidence analyses.

Thank you for the detailed response. The additional discussion on handling errors from the upstream model was helpful. The end-to-end latency analysis also clarified my earlier concerns. I have no further questions and will keep my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary for the Area Chair: Revisions and Discussion Outcome

Dear Area Chair,

Thank you for considering our submission and for the opportunity to address the reviewers' concerns during rebuttal. Their feedback helped us explain the method more clearly in the manuscript and strengthen the evaluation with additional experiments.

(1) CAUTION's Core Advantage

In short-video fake news detection, some videos can be judged reliably using only their own multimodal cues. Others can be internally coherent yet factually wrong; verifying them requires external evidence, sometimes over several rounds. CAUTION adapts the depth of verification to each video: it decides whether to seek external evidence, when to stop checking, and whether the evidence warrants changing the initial prediction.

On FakeSV (542 test videos), CAUTION correctly classifies 37 more videos than Fast-only while using 56.51% less evidence than the all-evidence baseline. On FakeTT, using all available evidence lowers accuracy below Fast-only, whereas CAUTION modestly improves on Fast-only while using 77.24% fewer evidence units and making fewer harmful revisions than the all-evidence baseline. These results support the value of sample-adaptive evidence reasoning over either uniform strategy, balancing verification effectiveness, reliability, and computational cost.

(2) Revisions and New Evidence from the Rebuttal

Reviewer iRhL: A new latency experiment and clearer safeguards. We clarified how gating and relation checks limit the influence of unreliable evidence. We ran a new timing experiment, comparing all three policies on the same 50 test videos per dataset. CAUTION reduced mean per-video latency by 61.4% on FakeSV and 67.0% on FakeTT versus the all-evidence baseline (Appendices I and J).

Reviewer 3JxN: The comparisons separate the roles of routing and stopping. Relative to conformal-only stopping, adding preservation stopping saved evidence on both datasets without changing final predictions. We also reported errors by exit branch, audited source provenance, and clarified the evidence inputs for each baseline.

Reviewer Tzx1: Improved readability and a detailed worked example. We turned the example that Reviewer Tzx1 found helpful into Figure 3 (Sec. 2.3, p. 6). In one figure, readers see why CAUTION keeps checking when the first evidence round raises confidence in the wrong label, and how the next round exposes the location error and lets the system stop. We also explain the key symbols at first use and correct the affected baseline metrics in Table 2.

Class-wise coverage and consistency across FakeTT runs. We added class-wise coverage checks and a calibration sensitivity experiment on the chronological test splits. We found that recalibration can narrow the coverage gap without retraining either predictor. Lowering α\alpha from 0.05 to 0.03 raised coverage for later fake videos from 88.89% to 95.96% before evidence and from 90.43% to 95.74% across all evidence rounds. We also evaluated two additional FakeTT configurations. All three configurations correctly classified two more videos than their respective Fast-only counterparts and matched or exceeded the strongest baseline, D²et (83.95%).

Discussion outcome. In the follow-up discussion, all three reviewers confirmed that our responses addressed their questions. The reviewer ratings changed from 3/3/2 to 3/3/3. The discussion helped us make the manuscript easier to read and deepen our experimental analysis. We sincerely appreciate the time and care you and the reviewers devoted to the review and discussion. We hope this brief summary gives you a quick overview of the revisions and additional evidence.

AuthorsOpenReview ↗

Summary for the Area Chair: Revisions and Discussion Outcome

For the locations of these changes in the revised manuscript, see Table A.1.

Table A.1. Key reviewer concerns and how the revised manuscript addresses them.

\hline \textbf{Reviewer concern} & \textbf{What the revision shows} & \textbf{Where to find it} \\\\ \hline \begin{array}{l} \textbf{Can noisy evidence} \\\\ \textbf{cause a harmful flip?} \end{array} & \begin{array}{l} \text{Explains how gating and relation checks} \\\\ \text{constrain revisions, and reports ablations} \\\\ \text{that remove individual safeguards.} \end{array} & \begin{array}{l} \text{App. I, p. 22} \\\\ \text{Table 11} \end{array} \\\\ \hdashline \begin{array}{l} \textbf{Does selective checking} \\\\ \textbf{save time?} \end{array} & \begin{array}{l} \text{Measures per-video latency} \\\\ \text{and breaks it down by stage.} \end{array} & \begin{array}{l} \text{App. J, p. 23} \\\\ \text{Tables 12–13} \end{array} \\\\ \hdashline \begin{array}{l} \textbf{How do the stopping} \\\\ \textbf{rules differ?} \end{array} & \begin{array}{l} \text{Distinguishes a singleton check from} \\\\ \text{preservation, which estimates agreement} \\\\ \text{with the conformal-only decision.} \end{array} & \begin{array}{l} \text{Sec. 2.3, pp. 5–6} \\\\ \text{App. C, p. 15} \end{array} \\\\ \hdashline \begin{array}{l} \textbf{How does CAUTION make} \\\\ \textbf{and revise a decision?} \end{array} & \begin{array}{l} \text{Adds a worked example covering initial} \\\\ \text{prediction, evidence gathering, revision,} \\\\ \text{and stopping.} \end{array} & \begin{array}{l} \text{Fig. 3, p. 6; Secs. 2.1–2.3} \\\\ \text{Apps. G–H, pp. 20–21} \end{array} \\\\ \hdashline \begin{array}{l} \textbf{How well does calibration} \\\\ \textbf{hold on later videos?} \end{array} & \begin{array}{l} \text{Reports class-wise test coverage} \\\\ \text{and how calibration affects accuracy} \\\\ \text{and evidence use.} \end{array} & \begin{array}{l} \text{App. F.1, pp. 18–19} \\\\ \text{Tables 6–8} \end{array} \\\\ \hdashline \begin{array}{l} \textbf{Are baseline metrics} \\\\ \textbf{comparable?} \end{array} & \begin{array}{l} \text{Defines two-class Macro-F1 and} \\\\ \text{corrects the affected Qwen-VL and} \\\\ \text{FakingRecipe entries.} \end{array} & \begin{array}{l} \text{Sec. 4.1, p. 7} \\\\ \text{Table 2, p. 8} \end{array} \\\\ \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.