- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper introduces CAUTION, a method for detecting fake news in short videos. It first makes an initial prediction using the video’s text, images, and audio. A conformal prediction rule then decides which videos need further checking. For these cases, the system examines external evidence for individual claims, adds the findings to a graph, and updates its prediction. A learned gate controls how much the evidence changes the initial decision, while a stopping rule decides whether more evidence is needed. The authors also build SV-CoRE, a repository of claims and evidence analyses. Experiments on FakeSV and FakeTT compare accuracy, evidence usage, and different components of the method. The reported results show that selective checking can improve accuracy while using fewer evidence analyses than processing all available evidence.
Critical strengths
-
I find the focus on deciding when additional evidence is useful particularly worthwhile. The method addresses both the cost of further checking and the risk of changing an initially correct prediction. Combining selective evidence gathering with controlled updates gives the approach a clear practical purpose: spend more effort on difficult cases while limiting unnecessary revisions.
-
The experiments examine how often evidence corrects a mistake or harms an initially correct prediction, alongside accuracy and evidence usage. These measurements help explain why selective checking can be useful. Comparisons with fast-only prediction, full-evidence processing, and confidence-based routing, together with the component ablations, provide useful support for the design. The fixed SV-CoRE evidence repository also provides a basis for repeatable comparisons.
Critical weaknesses
The main idea is understandable, but I found the method section difficult to follow, especially how the initial prediction, evidence gathering, and later revisions fit together. Several components are introduced with little explanation of their purpose before the equations. For example, appears in Equation (3) before its role is explained, while and require more explicit descriptions.
I also struggled with the conformal routing and reliability-gated revision sections. The mathematical descriptions are provided, but the intuition behind the thresholds, counterfactual graph, and correction gate is harder to follow. Moving between the main text and appendices makes it difficult to keep track of the complete process. A short example would help explain how these components work together and why each is needed.
The metric reporting in Table 2 needs clarification. For Qwen-VL on FakeTT, the reported accuracy is 58.86%, while macro-F1 is 73.89%. Under the standard binary definition, macro-F1 cannot exceed accuracy when both are computed from the same predictions. This could be a labeling issue or a difference in how the metrics were calculated, but it raises a concern about whether the F1 scores are directly comparable across methods. The authors should clarify the averaging convention and verify that it is applied consistently across the reported results.
The controller is a key part of the method, but I could not find results showing how often its set of possible labels includes the correct answer, separately for real and fake videos. It would help to report this and show how changing the allowed error level affects accuracy and the amount of evidence used. These checks are especially useful because the rule is calibrated using older videos and tested on newer ones. The theoretical guarantee assumes these groups follow similar statistical patterns, which may become less reliable as video content changes over time.
Questions for authors
-
I was confused by the Qwen-VL results on FakeTT: accuracy is 58.86%, while macro-F1 is 73.89%. Could you explain how these were calculated? Is the F1 score for one class or an average over both, and is the same definition used for all methods?
-
Could you show how often the conformal prediction sets include the correct label for real and fake videos separately? I would like to see how these rates compare with the intended coverage on the chronological test splits.
-
Could you walk through one example to explain how the initial prediction, evidence gathering, revision, and stopping steps fit together, and why each component is needed? Clearer explanations of all terms, e.g. , , when they first appear would also make the method easier to follow.
-My main reservation is that the controller’s calibration is a central contribution, yet its empirical coverage is not reported. Together with the unresolved metric definitions, this leaves me insufficiently convinced by the current evaluation. Clarifying these would be helpful for me.
10 comments
Weakness 4 & Question 2 (Part 1/3): Chronological Splits
Thank you for pointing out that changes in video content between the earlier calibration set and the later test set could affect how well the rule works. Your comment prompted us to measure how often the prediction sets included the correct label on the later test splits. Below, we document the publication-date ranges for the chronological splits (Table R3.1). We then compare coverage for real and fake videos with the intended level, both before using any evidence and across all evidence rounds (Table R3.2). We also show how changing the allowed error level affected accuracy and how much evidence the system used (Table R3.3).
All models are trained, calibrated, and tested strictly in chronological order. Auditing the video publication timestamps (UTC+08:00) across the official splits confirms zero temporal overlap between calibration (validation) and test periods:
Table R3.1: Chronological publication-date horizons across official benchmark splits.
\hline \textbf{Benchmark (Train / Val / Test }N\textbf{)} & \textbf{Training} & \textbf{Validation / Calibration} & \textbf{Held-out test} \\\\ \hline \textbf{FakeSV}\text{ (2,536 / 546 / 542)} & \text{2017-10-08 – 2021-06-26} & \text{2021-06-26 – 2021-10-09} & \textbf{2021-10-09 – 2022-02-06} \\\\ \textbf{FakeTT}\text{ (1,393 / 299 / 299)} & \text{2019-05-06 – 2023-10-20} & \text{2023-10-20 – 2024-01-18} & \textbf{2024-01-18 – 2024-03-13} \\\\ \hline \end{array}$$Weakness 4 & Question 2 (Part 2/3): Class-Wise and Overall Empirical Coverage Across α
We evaluate empirical coverage (the fraction of test videos whose ground-truth label is included in the calibrated prediction set) at two levels:
- Round 0 (; / ): Whether before any external evidence is retrieved (evaluated on 100% of test videos).
- Evidence Rounds (; / ): Each added claim--evidence unit produces a revised prediction and prediction set. A video is covered only if its true label belongs to every prediction set across all available evidence rounds, including rounds beyond the actual stopping point. The 5 FakeSV and 11 FakeTT videos without an evidence prefix are excluded only from this coverage calculation and remain in all classification metrics.
Table R3.2: Empirical test coverage across allowed error levels on chronological test splits.
\hline & & & \textbf{Round 0 }(r=0) & & \hspace{0.3em} & & \textbf{Evidence Rounds }(r \ge 1) & \\\\ \textbf{Dataset} & \alpha & \textbf{Fake} & \textbf{Real} & \textbf{Overall} & & \textbf{Fake} & \textbf{Real} & \textbf{Overall} \\\\ \hline \textbf{FakeSV} & 0.03 & 99.67\\% & 94.96\\% & 97.60\\% & & 99.67\\% & 94.87\\% & 97.58\\% \\\\ & \mathbf{0.05} & \mathbf{99.01\\%} & \mathbf{93.28\\%} & \mathbf{96.49\\%} & & \mathbf{99.34\\%} & \mathbf{94.44\\%} & \mathbf{97.21\\%} \\\\ & 0.07 & 99.01\\% & 86.97\\% & 93.73\\% & & 99.01\\% & 91.45\\% & 95.72\\% \\\\ & 0.10 & 97.70\\% & 80.25\\% & 90.04\\% & & 97.69\\% & 88.03\\% & 93.48\\% \\\\[3pt] \textbf{FakeTT} & 0.03 & 95.96\\% & 100.0\\% & 98.66\\% & & 95.74\\% & 100.0\\% & 98.61\\% \\\\ & \mathbf{0.05} & \mathbf{88.89\\%} & \mathbf{100.0\\%} & \mathbf{96.32\\%} & & \mathbf{90.43\\%} & \mathbf{100.0\\%} & \mathbf{96.88\\%} \\\\ & 0.07 & 86.87\\% & 96.50\\% & 93.31\\% & & 86.17\\% & 96.91\\% & 93.40\\% \\\\ & 0.10 & 83.84\\% & 91.00\\% & 88.63\\% & & 86.17\\% & 90.21\\% & 88.89\\% \\\\ \hline \end{array}$$ * **Evaluating the reviewer's temporal-shift concern:** At our default $\alpha=0.05$, while aggregate coverage exceeds 95% (`96.32%`–`97.21%`), **class-wise empirical coverage is asymmetric**: `FakeSV-Fake` (`99.01%` Round 0) and `FakeTT-Real` (`100.0%` at Round 0 and across all evidence rounds) exceed 95%, whereas `FakeSV-Real` (`93.28%` at Round 0; `94.44%` across evidence rounds) and `FakeTT-Fake` (`88.89%` at Round 0; `90.43%` across evidence rounds) fall below the 95% nominal level. To examine this potential temporal shift, we compare the nonconformity-score distributions between calibration and test splits. On FakeTT-Fake at $\alpha=0.05$, the fraction of Round-0 scores exceeding $\widehat q_{\text{Fake}}^{(0)}$ increases from 4.94% (4/81) on the validation calibration split to 11.11% (11/99) on the later test split. This difference is consistent with a possible temporal distribution shift, alongside finite-sample variability. * **Effect of adjusting $\alpha$:** Lowering $\alpha$ increases empirical coverage by making the conformal prediction sets more inclusive, without retraining either prediction model. In particular, reducing $\alpha$ from $0.10$ to $0.03$ raises Round-0 coverage by 14.71 pp on FakeSV-Real (80.25% → 94.96%) and 12.12 pp on FakeTT-Fake (83.84% → 95.96%).
Weakness 3 & Question 1: Qwen-VL F1 Correction
Thank you for identifying this reporting error. The intended metric in Table 2 is Macro-F1, the unweighted mean of the Fake-class and Real-class F1 scores. The Qwen-VL row mistakenly listed Fake-class F1 on FakeSV (0.7635) and Real-class F1 on FakeTT (0.7389). The original Precision and Recall were also class-specific. The corrected row uses two-class macro averages for F1, Precision, and Recall, based on the reported metrics for the complete original test sets (FakeSV: 542 videos; FakeTT: 299 videos).
\qquad F_{1,c}=\frac{2TP_c}{2TP_c+FP_c+FN_c}.$$ $$\begin{array}{lrrrrrrrr} \hline & & \rlap{\kern-1em\textbf{FakeSV}} & & & & \rlap{\kern-1em\textbf{FakeTT}} & & \\\\ \textbf{Model} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Precision} & \textbf{Recall} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Precision} & \textbf{Recall} \\\\ \hline \text{Qwen-VL} & 0.7417 & 0.7395 & 0.7388 & 0.7415 & 0.5886 & 0.3852 & 0.3567 & 0.4451 \\\\ \hline \end{array}$$ While checking Table 2 of the original manuscript, we also found that the FakingRecipe entry on FakeSV reported weighted F1 (0.8192) instead of macro-F1 (0.8144). We will correct the Qwen-VL row and the FakeSV FakingRecipe entry in the revised Table 2. Thank you again for pointing this out.