- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldFact-Bench, a benchmark for evaluating whether multimodal models can verify the consistency between image content and real-world facts or rules, going beyond traditional image forensics that mainly focuses on image-internal artifacts and visual plausibility. The benchmark contains 1,274 source-aligned real-fake image pairs across four verification regimes and ten semantic domains. Each fake image is created by introducing a specific factual conflict while preserving other visual content, enabling evaluation of whether models can identify subtle inconsistencies without being provided with a predefined verification target or reference evidence. The paper further proposes PERSIST-Agent, an iterative verification framework built around a persistent verification state. The agent maintains structured records linking candidate facts, visual observations, retrieved evidence, and verification statuses, and uses this state to guide subsequent inspection, retrieval, and decision-making. Extensive experiments evaluate existing forensic detectors, multimodal models, retrieval-augmented approaches, and PERSIST-Agent. The results show that current models suffer from strong label biases and limited pair-wise discrimination, while PERSIST-Agent improves over direct judgment and retrieval baselines by maintaining verification state throughout the reasoning process. Overall, the paper presents an interesting benchmark and agent framework for studying image-world consistency verification, highlighting an important gap between visual plausibility and factual correctness in multimodal systems.
Critical strengths
- Novel and important task formulation. The paper introduces a new perspective beyond conventional image forensics by evaluating whether visual content is consistent with real-world facts and rules. This is an important direction as generated images become increasingly realistic and may no longer contain obvious visual artifacts.
- Carefully constructed benchmark. WorldFact-Bench is thoughtfully designed with source-aligned real-fake pairs, controlled factual conflicts, and multiple verification regimes. The construction aims to isolate factual inconsistency while preserving visual plausibility.
- Reasonable agent formulation. The persistent verification state is a reasonable design choice for multi-step verification. Maintaining candidate facts, observations, evidence, and verification status helps avoid losing intermediate reasoning information.
- Analysis beyond final accuracy. The paper provides detailed diagnostic analysis, including oracle studies, component ablations, failure analysis, and pair-wise prediction analysis.
Critical weaknesses
- Limited benchmark scale and coverage. Although WorldFact-Bench introduces an interesting task, the dataset size remains relatively limited for a general multimodal benchmark. With only 1,274 source-aligned image pairs across multiple domains, it is unclear whether the benchmark sufficiently captures the diversity of real-world factual inconsistencies.
- Limited real-world validation. The benchmark focuses on controlled factual edits generated from source images. However, real-world misinformation images often involve complex combinations of editing, external context, and misleading narratives. More evaluation on naturally occurring misinformation or web-scale images would strengthen the contribution.
- The proposed agent improves performance but does not fully solve the task. PERSIST-Agent provides consistent improvements over direct judgment and retrieval-augmented baselines, but the absolute performance remains low. For example, the best reported Pair accuracy is still only around 32%, indicating that reliable image-world consistency verification remains far from solved.
- The methodological novelty of PERSIST-Agent is moderate. The proposed persistent verification state is useful, but the overall framework is closely related to existing tool-augmented reasoning agents. The authors should further clarify what fundamental differences distinguish PERSIST-Agent from existing memory-based or stateful agent frameworks.
Questions for authors
- How well does WorldFact-Bench transfer to naturally occurring misinformation images?
- How does PERSIST-Agent compare with stronger tool-using multimodal agents?
- Which stage contributes most to the remaining failures: visual understanding, retrieval, or reasoning?
- Can the authors scale WorldFact-Bench to more diverse real-world factual conflicts?
2 comments
Response to Reviewer UvLz (Part 2/2)
Q4. Differences from existing memory-based or stateful agents, and comparisons with stronger tool-using agents
Memory-based or stateful agents often retain execution histories, summaries, or reflections to reuse past observations and actions. PERSIST-Agent organizes its state around candidate facts: each candidate links a visual observation, applicable evidence, an evidence-supported value, and a verification status. The distinction lies in how the state is organized, what its fields mean, and how it guides the next action.
This design provides three benefits:
- Keeping evidence linked to the right target. When switching between candidates, the agent retains which evidence applies to which entity, attribute, and context.
- Preserving the basis for factual comparison. The observed value and the evidence-supported value are recorded separately. This helps prevent retrieved expectations from replacing what was actually observed in the image.
- Making further verification needs explicit. Incomplete or ambiguous candidates remain available for further inspection, refined queries, or additional evidence. New information can revise a judgment while preserving progress on other candidates.
The state definitions and instructions are provided in Appendix C.1, Table 13, and C.4.
We compare this design with a full-history agent that retains multi-round search and all prior observations and tool results. It also receives independent prompt optimization under the same budget.
The state table improves Pair from 16.12% to 18.46% on Qwen3-VL-8B and from 14.96% to 18.27% on InternVL3.5-8B, as reported in Appendix D.3, Table 16. These results show that explicitly organizing verification state around candidate facts remains useful even when the complete history is already available.
Appendix D.4, Table 17 further evaluates the framework on stronger backbones. The full-history comparison and scaling experiments examine the effects of state design and backbone capability, respectively. They do not replace direct comparisons with specific external tool-using agents, which remain a useful direction for further evaluation.
Q5. Which stages account for the remaining failures?
The added trace analysis identifies visual understanding/target discovery and evidence reasoning/decision as the main remaining difficulties.
We examine all incorrect single-image predictions from one fixed test run of PERSIST-Agent with Qwen3-VL-8B. Each trace is compared with the construction records, including the entity, target fact, visible value, and applicable evidence.
Appendix D.5, Table 18 reports the following distribution:
| Primary failure stage | Share |
|---|---|
| Visual understanding and target discovery | 51.68% |
| Evidence retrieval | 11.63% |
| Evidence reasoning and decision | 36.69% |
Each erroneous trace receives one primary label: the earliest relevant error that remains uncorrected and leads to the final wrong prediction. The first category includes missed verification targets and incorrect visual readings. Applying evidence to the wrong entity, version, or context is classified as a reasoning error.
These findings suggest that additional retrieval alone is insufficient. Models also need to identify and correctly observe the decisive target, then apply the evidence to the image accurately. This provides concrete directions for further improvement.
Response to Reviewer UvLz (Part 1/2)
We thank the reviewer for recognizing the importance of the task, the source-aligned pair design, and the persistent verification state. We have added construction and review details, larger-backbone evaluations, an independently optimized full-history baseline, and trace-based failure analysis.
Q1. Limited dataset size and coverage
We agree that WorldFact-Bench has limited size and coverage. Our goal is to establish a controlled, traceable evaluation setting for image–world consistency, rather than claim complete coverage of real-world factual conflicts.
Building each sample requires more than producing a plausible edit. The target fact must be supported by evidence that applies to the depicted entity and context. The edit must introduce a clear factual conflict, rather than a valid variant, while preserving identification cues, non-target content, and visual plausibility. These requirements make expansion a sample-by-sample verification process.
Appendix A.3–A.4 details the construction and filtering procedure. From 5,600 initial candidates, 1,274 pairs were retained. Three reviewers independently assess each candidate at every filtering stage, and all three must approve it.
The current version covers four verification regimes, ten semantic domains, and eight manipulation types. It provides a shared test set for studying target discovery, factual verification, and pair discrimination. We interpret the results within this scope and clarify the boundaries in Appendix F.
Further expansion can follow the construction procedure and 221 atomic edit recipes described in Appendix A.3, Table 6. The dataset release will include sources, target facts, reference evidence, and edit instructions to support expansion under the same criteria.
Q2. Transfer to naturally occurring misinformation images
Naturally occurring misinformation is an important evaluation direction. Transfer performance requires a separate test with clearly defined inputs and factual evidence.
Controlled edits let us identify the exact factual change while keeping other content largely unchanged. This helps test whether a model responds to that change.
Naturally occurring misinformation can involve different mechanisms. Some images contain factual conflicts within their visual content. Others become misleading through an incorrect caption, event context, or surrounding narrative. For the latter, the image alone may not provide enough information to determine whether it is misleading.
WorldFact-Bench focuses on discovering verification targets from a single image and checking its content against real-world facts. Appendix F explains this scope. The current results support evaluation on controlled factual conflicts; they do not directly establish transfer accuracy on naturally occurring misinformation. Extending the evaluation to such cases with verifiable evidence is an important next step.
Q3. Limited absolute Pair accuracy
We agree that the current method does not yet provide reliable image factual verification. The paper establishes an evaluation setting and demonstrates initial progress on this capability.
Detecting generation artifacts, checking image–text agreement, and verifying image content against real-world facts address different questions. A visually plausible image can still contain a factually incorrect attribute or relation. Our setting requires models to discover what to verify, rather than assess a supplied claim.
PERSIST-Agent outperforms direct judgment and retrieval augmentation on both 8B backbones. We have extended the evaluation to seven backbone configurations in Appendix D.4, Table 17.
Across all seven backbones, the gains are 8.51–14.09 percentage points over direct judgment and 3.64–8.21 points over retrieval augmentation.
These results show that stronger backbones and the verification procedure can improve performance, while the task remains unresolved. The pair outcomes, evidence comparisons, and failure analysis help identify specific capabilities for future improvement.