acceptodds
Under review as a conference paper at ICLR 2027

Verifiability, Not Confidence: Cheap Multimodal Rumor Detection That Knows When to Refuse

Abstract

Multimodal misinformation detection faces a practical trade-off, in which discriminative detectors are cheap but less accurate while agentic pipelines are accurate but costly. We propose a minimal B vision-language classifier that answers each post in one forward pass, without an agent or retrieval. It approaches recently published detectors on AMG, FineFake, and Weibo at a measured USD per post. Auditing the residual errors reveals a deeper failure: many posts assert no verifiable claim, yet on content the detector never models its confidence is at its highest. We call this property *verifiability* and show that, under a fixed judge rubric, it is linearly decodable from frozen MLLM representations, with AUROC on AMG (in-domain), FineFake, and Weibo (both zero-shot transfers), and that the same readout transfers from English to Chinese and across model families down to B parameters while remaining near-orthogonal to confidence. A verifiability gate restores rejection where confidence fails, and a cheap resolver behind it keeps the cascade at or above direct accuracy. We believe verifiability offers a principled coordinate for abstention in multimodal detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Rumor Detection, Selective Prediction, Verifiability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C6RsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selective multimodal rumor detection using a 4B vision–language classifier and a linear probe that predicts whether a post contains a verifiable claim. The probe is trained with LLM-generated labels under a fixed rubric and combined with detector confidence to route selected posts to a second-level handler. Experiments cover AMG, FineFake, Weibo, and MR2, including cross-language transfer, human label audits, alternative rejection signals, and cost analysis. The probe achieves strong label prediction, and the gate improves MR2 answered accuracy at matched coverage.

Critical strengths
  1. Substantial evidence for an inexpensive claim-presence signal. Cross-domain and English-to-Chinese transfer, human audits, and lexical and label-swap controls support the usefulness of the linear probe under the stated rubric.

  2. Useful analysis of selective prediction. The paper examines coverage, intercepted errors, and the cost of rejecting correct predictions. On MR2, the gate improves answered accuracy by 2.8 points at approximately 95% coverage, with a reported confidence interval of [1.8, 3.7].

Critical weaknesses
  1. The main ablation mixes detector configurations. Section 5.3 states that all variants use the same detector. Appendix M instead explains that Table 2’s full-system result of 0.812 uses DO-LoRA, while the self-contained probe comparison uses another detector configuration. The matched gate-only improvement is 0.035, compared with the displayed difference of 0.054. Table 2 therefore needs a common-detector comparison to establish the contribution of the gate.

  2. Cascade accuracy and cost are reported at different operating points. The FineFake and Weibo evaluations reject 47.8% and 39.6% of posts, respectively. Table 28 prices these cascades using an assumed rejection rate of 8–20% borrowed from MR2. Although this extrapolation is disclosed, the resulting cost range does not establish the cost advantage at the reported accuracy. Both quantities should be measured using the same thresholds and handler configuration.

Questions for authors
  1. Can you provide a revised Table 2 with all variants evaluated using the same detector, and clearly distinguish judge-based gating from the self-contained linear probe?

  2. What are the measured end-to-end costs at the exact FineFake and Weibo operating points used for the reported cascade accuracies?

  3. Does the MR2 gating benefit persist on independently human-labeled claim/no-claim examples, including under the alternative rubric examined in Appendix G?

Reviewer_78xyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when multimodal rumor detectors should abstain rather than make a confident true-or-false prediction. The authors train a lightweight probe on a Qwen3-VL detector’s hidden representations to identify posts without verifiable factual claims. They combine this signal with the detector’s classification score through benchmark-specific gates, with rejected posts passed to a second-stage handler for further judgment. Experiments on AMG, FineFake, Weibo, and MR2 examine detection performance, selective prediction, transfer, and inference cost. The results suggest that the presence of a verifiable claim is recoverable from model representations and can provide useful information beyond confidence, although the appropriate gating policy and its benefits vary across datasets.

Critical strengths
  • The paper explores a useful distinction between how confident a detector is and whether the input contains a factual claim to check. Reading this second signal from existing hidden states adds little inference overhead, and the probe’s transfer from English datasets to Chinese Weibo is an encouraging result.

  • The experiments go well beyond reporting overall accuracy. The paper examines different backbones, representation layers, alternative signals, and human agreement with the annotations. It also reports cases where the approach is less effective, which helps clarify what the probe captures and where its usefulness is limited.

Critical weaknesses
  • I found the paper’s presentation difficult to follow, as the evaluation settings are described differently across sections. Line 140 says “Splits follow the original papers,” but Appendix L describes using “our own splits” (lines 1186–1187, 1230–1232). MR2 is also described as using “no online retrieval, no online LLM” (line 1235), while Appendix P identifies SenseNova and Bocha as its handler and retrieval backend (lines 1575–1577).

  • High no-claim scores lead to rejection on AMG but favor retaining rumor predictions on FineFake and Weibo. I understand the validation-based explanation in §4.2, but this means the same signal leads to opposite decisions depending on the benchmark. The probe may transfer, while the overall policy still needs labeled target-domain data to determine how to use it. This makes the method less general than the framing around verifiability-based abstention suggests.

  • Figure 2 compares the base detector’s accuracy and cost with other systems, leaving out the second-stage handler. The caption acknowledges this and the differences in evaluation protocols, but the comparison still does not show the complete method’s cost–performance tradeoff. Table 28 estimates the full cost using 8%–20% referral rates for FineFake and Weibo, even though Table 1 reports 47.8% and 39.6%. It is therefore difficult to tell how much the full system costs at the performance levels actually reported.

  • Simply prompting Qwen3-VL-4B to identify verifiable claims seems like a natural baseline, but I could not find this comparison. The frozen-backbone experiment still uses trained probes. Without testing prompted judgments in the same gating pipeline, it is difficult to assess the probe’s added value in effectiveness and cost.

Questions for authors
  • Could you clarify the conflicting descriptions of the data splits and MR2’s use of online models and retrieval? A consolidated account of the main experimental settings would help.

  • Section 4 initially presents a common two-gate rule, but the gate direction later varies by dataset. How should this dependence on labeled target-domain validation data limit the claimed generality of the method?

  • Would the routing gains hold on unseen topics or later time periods without recalibrating the gate direction and thresholds? This would help establish whether the complete method generalizes beyond the benchmark-specific validation settings.

  • Could you report full-system costs at the actual referral rates and operating points used in the main experiments, including the basis for model and retrieval charges? This would make the cost–performance comparison easier to interpret.

  • How does the method compare with directly prompting the base Qwen3-VL-4B to identify verifiable claims and using those judgments in the same pipeline? Some prompt tuning may be needed to make this a fair comparison.

Reviewer_t8aROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Confidence-based rumor detectors can assign high confidence to posts that contain no factual claim, even though such posts do not fit a real-versus-rumor decision. This paper proposes verifiability as an additional signal for handling this issue. The method trains a low-cost multimodal detector, extracts a linear verifiability signal from frozen representations, and combines it with confidence to route selected posts to a second-stage handler. Experiments show that the signal transfers across datasets and languages and identifies no-claim content missed by confidence alone.

Critical strengths

1 A meaningful limitation of confidence. This paper shows that confidence within a binary classification task does not necessarily identify inputs outside that task, motivating a separate signal for whether a prediction is appropriate. 2 A lightweight detection mechanism. The proposed linear probe extracts the additional signal from frozen representations, avoiding a separate retrieval process for the initial scope decision. 3 Broad empirical analysis. Experiments examine cross-domain and cross-language transfer, multiple model families, and lexical and novelty baselines, providing useful evidence that the signal is not simply a confidence score.

Critical weaknesses

1 The paper conflates verifiability with mere claim detection. The annotation rubric only checks for the presence of a factual claim, not whether external evidence exists to resolve it. No direct comparison is provided to distinguish this construct from established claim detection or check-worthiness tasks.

2 The probe learns spurious correlations from LLM-generated labels rather than a stable notion of verifiability. An independently authored rubric achieves only moderate agreement with the original labels, indicating that the model’s strong transfer reflects shared LLM labeling conventions, not generalizable human judgment.

3 The no-claim gate behaves contradictively across domains: high scores trigger rejection on AMG but reinforce rumor predictions on FineFake and Weibo. Because the decision direction is tuned using target-domain labels, the mechanism cannot be interpreted as a universal, verifiability-driven rejection rule.

4 The comparison is incomplete and biased. Rejection is only evaluated on rumor-side predictions, and precision gains come at the cost of substantial coverage loss. No matched-coverage analysis or comparison against a baseline with an explicit no-claim class is provided to validate the true benefit of the proposed gate.

5 The paper would benefit from further polishing to improve clarity and readability.

Questions for authors

1 Could the authors clarify how verifiability differs from claim detection and compare against direct claim-detection baselines?

2 How stable is the learned signal under independently designed human annotation rubrics, and can one rejection policy transfer without target-domain labels?

3 Does the method remain better than confidence gating and a detector at matched coverage when both predicted classes can be rejected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.