- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper models multimodal fact verification as a closed-loop sequential decision problem for an agent and proposes the VeriLoop framework. The framework builds a proposition-level verification graph, continuously maintains an explicit verification state, autonomously chooses propositions to verify, gathers text/image cross-modal evidence, updates the state, and decides when to stop; it separates local verification of single propositions from deterministic assertion-based conclusion derivation, ensuring conclusions are traceable. Experiments on the Factify, MOCHEG, and VERITE datasets show that the effectiveness of proposition decomposition verification highly depends on evidence quality—clean evidence improves performance, while noisy retrieval actually harms it, and this effect cannot be reproduced using chain-of-thought reasoning. Verification gains are concentrated in the early steps, and more verification actions do not necessarily lead to better results; stopping strategies, conclusion combination rules, and label granularity are interdependent. The paper points out that multimodal fact verification is not just about reasoning based on evidence but also about controlling the entire sequential verification process.
Critical strengths
-
Novel problem formulation framing multimodal fact‑verification as closed‑loop sequential decision‑making Most prior work treats fact‑checking as one‑shot or multi‑hop reasoning given evidence. Instead, VeriLoop explicitly models mutable proposition‑level verification state, targeted evidence acquisition, replanning and stopping. Separating local proposition verification from deterministic claim‑level verdict derivation brings full traceability for every final prediction. This agentic control perspective opens a new research direction for fact‑verification.
-
Empirically rich, careful empirical investigation yielding actionable insights for the community Beyond proposing VeriLoop architecture, the paper systematically studies when structured per‑proposition resolution helps/hurts, diminishing returns of repeated verification attempts, and dangerous benchmark artefacts in Factify / VERITE. The finding “more verification steps ≠ better performance” is counter‑intuitive and well‑supported by logged trajectories and budgeted offline replays; benchmark‑integrity diagnostics serve as important warning for future multimodal fact‑verification benchmark development.
Critical weaknesses
-
Key stopping‑policy conclusion relies entirely on offline counterfactual truncation without live online validation The promising macro‑F1 improvement under “assertion‑settled stopping” is obtained by cutting pre‑recorded trajectories post‑hoc, not by running the agent live with that stopping condition. As authors note, we cannot confirm whether this stopping rule would generalise to live execution, new datasets or benchmarks supporting the (MISLEADING) label. Without live agent rollouts, the practical utility of this stopping heuristic remains open.
-
Limited competitive leaderboard performance under zero‑shot setting, strong results depend on gold‑quality evidence VeriLoop obtains strong numbers mainly when gold reference evidence is provided. Under noisy retrieved evidence its per‑proposition resolution mechanism degrades performance (C5 vs C3 in Table2). On VERITE zero‑retrieved‑evidence zero‑shot setting, vanilla VeriLoop underperforms existing fine‑tuned VLMs. While the paper’s main contributions are insights about process control rather than pure leaderboard numbers, readers may expect clearer discussion about real‑world applicability when gold evidence is unavailable.
-
Poor figure visual quality hinders understanding core results Figures are over‑crowded, lack sufficient annotation, and mix distinct experimental paradigms (live trajectory / offline replay) without prominent visual differentiation. Several key curves and markers are hard to parse. This is a presentation deficit but impedes readers in grasping the paper’s central empirical take‑homes.
Questions for authors
-
In the assertion‑settled stopping counter‑factual, 473 verdicts change after early truncation. Could you elaborate on qualitative patterns of these changed samples? For instance: how many of these correspond to genuine cases that Factify label‑space cannot capture?
-
Would it be feasible in future work to run a small‑scale live online experiment with the assertion‑settled stopping rule, to test whether the offline‑observed gain transfers to live agent execution?
-
On Factify under noisy retrieval, per‑proposition resolution (C5) underperforms flattened reading (C3). Do you have intuitions on what kinds of retrieval errors cause this degradation? Are there simple mitigation directions?
-
VERITE results show that proposition decomposition helps, but role‑typing hurts performance. Could you give brief intuition why role typing brings negative effect on this benchmark?
-
Could you substantially revise figures: add clearer legends, explicitly mark which panels correspond to logged live trajectories vs offline frozen replay, and reduce over‑crowding? Many core claims hinge upon these plots.