acceptodds
Under review as a conference paper at ICLR 2027

WorldFact-Bench: Beyond Image-Internal Plausibility to Image-World Consistency

Abstract

Advances in image generation have made visual authenticity increasingly difficult to assess. Although image forensics now examines both generation artifacts and higher-level visual inconsistencies, a plausible image can still contradict real-world facts or rules. We introduce WorldFact-Bench to evaluate image-world consistency from a single image, without a predefined claim or verification target. The benchmark contains 1,274 source-aligned real–fake pairs across four verification regimes and ten semantic domains. Each pair introduces a specific, evidence-supported factual conflict while seeking to preserve non-target content and visual plausibility. Images are evaluated independently, and pair accuracy requires both members of a pair to be classified correctly. We further propose PERSIST-Agent, which organizes iterative verification around a persistent state linking candidate facts, visual observations, evidence, and verification statuses. This state guides subsequent inspection and retrieval while retaining unresolved candidates. With backbone weights fixed, harness self-optimization refines the agent’s prompts and execution rules through validation feedback. Experiments reveal strong label biases in several detectors and uneven gains from retrieval. On the evaluated 8B backbones, PERSIST-Agent improves pair accuracy over both direct judgment and retrieval-augmented baselines, while ablations support the role of persistent verification state. These findings highlight the value of state-guided verification and the remaining gap between visual plausibility and factual correctness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Image Forensics, Multimodal Fact-Checking

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_UvLzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldFact-Bench, a benchmark for evaluating whether multimodal models can verify the consistency between image content and real-world facts or rules, going beyond traditional image forensics that mainly focuses on image-internal artifacts and visual plausibility. The benchmark contains 1,274 source-aligned real-fake image pairs across four verification regimes and ten semantic domains. Each fake image is created by introducing a specific factual conflict while preserving other visual content, enabling evaluation of whether models can identify subtle inconsistencies without being provided with a predefined verification target or reference evidence. The paper further proposes PERSIST-Agent, an iterative verification framework built around a persistent verification state. The agent maintains structured records linking candidate facts, visual observations, retrieved evidence, and verification statuses, and uses this state to guide subsequent inspection, retrieval, and decision-making. Extensive experiments evaluate existing forensic detectors, multimodal models, retrieval-augmented approaches, and PERSIST-Agent. The results show that current models suffer from strong label biases and limited pair-wise discrimination, while PERSIST-Agent improves over direct judgment and retrieval baselines by maintaining verification state throughout the reasoning process. Overall, the paper presents an interesting benchmark and agent framework for studying image-world consistency verification, highlighting an important gap between visual plausibility and factual correctness in multimodal systems.

Critical strengths
  1. Novel and important task formulation. The paper introduces a new perspective beyond conventional image forensics by evaluating whether visual content is consistent with real-world facts and rules. This is an important direction as generated images become increasingly realistic and may no longer contain obvious visual artifacts.
  2. Carefully constructed benchmark. WorldFact-Bench is thoughtfully designed with source-aligned real-fake pairs, controlled factual conflicts, and multiple verification regimes. The construction aims to isolate factual inconsistency while preserving visual plausibility.
  3. Reasonable agent formulation. The persistent verification state is a reasonable design choice for multi-step verification. Maintaining candidate facts, observations, evidence, and verification status helps avoid losing intermediate reasoning information.
  4. Analysis beyond final accuracy. The paper provides detailed diagnostic analysis, including oracle studies, component ablations, failure analysis, and pair-wise prediction analysis.
Critical weaknesses
  1. Limited benchmark scale and coverage. Although WorldFact-Bench introduces an interesting task, the dataset size remains relatively limited for a general multimodal benchmark. With only 1,274 source-aligned image pairs across multiple domains, it is unclear whether the benchmark sufficiently captures the diversity of real-world factual inconsistencies.
  2. Limited real-world validation. The benchmark focuses on controlled factual edits generated from source images. However, real-world misinformation images often involve complex combinations of editing, external context, and misleading narratives. More evaluation on naturally occurring misinformation or web-scale images would strengthen the contribution.
  3. The proposed agent improves performance but does not fully solve the task. PERSIST-Agent provides consistent improvements over direct judgment and retrieval-augmented baselines, but the absolute performance remains low. For example, the best reported Pair accuracy is still only around 32%, indicating that reliable image-world consistency verification remains far from solved.
  4. The methodological novelty of PERSIST-Agent is moderate. The proposed persistent verification state is useful, but the overall framework is closely related to existing tool-augmented reasoning agents. The authors should further clarify what fundamental differences distinguish PERSIST-Agent from existing memory-based or stateful agent frameworks.
Questions for authors
  1. How well does WorldFact-Bench transfer to naturally occurring misinformation images?
  2. How does PERSIST-Agent compare with stronger tool-using multimodal agents?
  3. Which stage contributes most to the remaining failures: visual understanding, retrieval, or reasoning?
  4. Can the authors scale WorldFact-Bench to more diverse real-world factual conflicts?
2 comments
AuthorsOpenReview ↗

Response to Reviewer UvLz (Part 1/2)

We thank the reviewer for recognizing the importance of the task, the source-aligned pair design, and the persistent verification state. We have added construction and review details, larger-backbone evaluations, an independently optimized full-history baseline, and trace-based failure analysis.

Q1. Limited dataset size and coverage

We agree that WorldFact-Bench has limited size and coverage. Our goal is to establish a controlled, traceable evaluation setting for image–world consistency, rather than claim complete coverage of real-world factual conflicts.

Building each sample requires more than producing a plausible edit. The target fact must be supported by evidence that applies to the depicted entity and context. The edit must introduce a clear factual conflict, rather than a valid variant, while preserving identification cues, non-target content, and visual plausibility. These requirements make expansion a sample-by-sample verification process.

Appendix A.3–A.4 details the construction and filtering procedure. From 5,600 initial candidates, 1,274 pairs were retained. Three reviewers independently assess each candidate at every filtering stage, and all three must approve it.

The current version covers four verification regimes, ten semantic domains, and eight manipulation types. It provides a shared test set for studying target discovery, factual verification, and pair discrimination. We interpret the results within this scope and clarify the boundaries in Appendix F.

Further expansion can follow the construction procedure and 221 atomic edit recipes described in Appendix A.3, Table 6. The dataset release will include sources, target facts, reference evidence, and edit instructions to support expansion under the same criteria.

Q2. Transfer to naturally occurring misinformation images

Naturally occurring misinformation is an important evaluation direction. Transfer performance requires a separate test with clearly defined inputs and factual evidence.

Controlled edits let us identify the exact factual change while keeping other content largely unchanged. This helps test whether a model responds to that change.

Naturally occurring misinformation can involve different mechanisms. Some images contain factual conflicts within their visual content. Others become misleading through an incorrect caption, event context, or surrounding narrative. For the latter, the image alone may not provide enough information to determine whether it is misleading.

WorldFact-Bench focuses on discovering verification targets from a single image and checking its content against real-world facts. Appendix F explains this scope. The current results support evaluation on controlled factual conflicts; they do not directly establish transfer accuracy on naturally occurring misinformation. Extending the evaluation to such cases with verifiable evidence is an important next step.

Q3. Limited absolute Pair accuracy

We agree that the current method does not yet provide reliable image factual verification. The paper establishes an evaluation setting and demonstrates initial progress on this capability.

Detecting generation artifacts, checking image–text agreement, and verifying image content against real-world facts address different questions. A visually plausible image can still contain a factually incorrect attribute or relation. Our setting requires models to discover what to verify, rather than assess a supplied claim.

PERSIST-Agent outperforms direct judgment and retrieval augmentation on both 8B backbones. We have extended the evaluation to seven backbone configurations in Appendix D.4, Table 17.

BackbonePERSIST-Agent Pair accuracy
Qwen3-VL-32B29.83%
InternVL3.5-38B31.46%
Qwen3-VL-235B-A22B43.27%

Across all seven backbones, the gains are 8.51–14.09 percentage points over direct judgment and 3.64–8.21 points over retrieval augmentation.

These results show that stronger backbones and the verification procedure can improve performance, while the task remains unresolved. The pair outcomes, evidence comparisons, and failure analysis help identify specific capabilities for future improvement.

AuthorsOpenReview ↗

Response to Reviewer UvLz (Part 2/2)

Q4. Differences from existing memory-based or stateful agents, and comparisons with stronger tool-using agents

Memory-based or stateful agents often retain execution histories, summaries, or reflections to reuse past observations and actions. PERSIST-Agent organizes its state around candidate facts: each candidate links a visual observation, applicable evidence, an evidence-supported value, and a verification status. The distinction lies in how the state is organized, what its fields mean, and how it guides the next action.

This design provides three benefits:

  • Keeping evidence linked to the right target. When switching between candidates, the agent retains which evidence applies to which entity, attribute, and context.
  • Preserving the basis for factual comparison. The observed value and the evidence-supported value are recorded separately. This helps prevent retrieved expectations from replacing what was actually observed in the image.
  • Making further verification needs explicit. Incomplete or ambiguous candidates remain available for further inspection, refined queries, or additional evidence. New information can revise a judgment while preserving progress on other candidates.

The state definitions and instructions are provided in Appendix C.1, Table 13, and C.4.

We compare this design with a full-history agent that retains multi-round search and all prior observations and tool results. It also receives independent prompt optimization under the same budget.

The state table improves Pair from 16.12% to 18.46% on Qwen3-VL-8B and from 14.96% to 18.27% on InternVL3.5-8B, as reported in Appendix D.3, Table 16. These results show that explicitly organizing verification state around candidate facts remains useful even when the complete history is already available.

Appendix D.4, Table 17 further evaluates the framework on stronger backbones. The full-history comparison and scaling experiments examine the effects of state design and backbone capability, respectively. They do not replace direct comparisons with specific external tool-using agents, which remain a useful direction for further evaluation.

Q5. Which stages account for the remaining failures?

The added trace analysis identifies visual understanding/target discovery and evidence reasoning/decision as the main remaining difficulties.

We examine all incorrect single-image predictions from one fixed test run of PERSIST-Agent with Qwen3-VL-8B. Each trace is compared with the construction records, including the entity, target fact, visible value, and applicable evidence.

Appendix D.5, Table 18 reports the following distribution:

Primary failure stageShare
Visual understanding and target discovery51.68%
Evidence retrieval11.63%
Evidence reasoning and decision36.69%

Each erroneous trace receives one primary label: the earliest relevant error that remains uncorrected and leads to the final wrong prediction. The first category includes missed verification targets and incorrect visual readings. Applying evidence to the wrong entity, version, or context is classified as a reasoning error.

These findings suggest that additional retrieval alone is insufficient. Models also need to identify and correctly observe the decisive target, then apply the evidence to the image accurately. This provides concrete directions for further improvement.

Reviewer_QuN5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldFact-Bench, a benchmark designed to evaluate whether models can identify inconsistencies between image content and real-world facts or rules, rather than relying solely on image-internal visual plausibility. The benchmark contains 1,274 source-aligned real–fake image pairs spanning four verification regimes, ten semantic domains, and eight manipulation types. The authors also propose PERSIST-Agent, an iterative verification framework that maintains a structured state linking candidate facts, visual observations, retrieved evidence, and verification statuses. Experiments with conventional image detectors and multimodal large language models (MLLMs) reveal substantial limitations in factual inconsistency detection. The proposed agent improves pair accuracy over direct judgment and retrieval-augmented baselines on the evaluated 8B backbones.

Critical strengths

1.Meaningful problem formulation. The paper addresses an important limitation of existing image authenticity assessment: visually plausible images may still contain factual inconsistencies that cannot be identified from image-internal evidence alone. Evaluating image-world consistency without providing a predefined verification target is a meaningful and challenging setting.

2.Well-structured benchmark design. WorldFact-Bench covers diverse semantic domains and verification requirements. The use of source-aligned real–fake pairs helps control differences in image content, while pair accuracy provides a useful complement to conventional classification accuracy by exposing one-sided prediction biases.

3.Useful diagnostic experiments. The paper evaluates a range of specialized detectors and MLLMs and includes oracle experiments, component ablations, retrieval diagnostics, and failure analyses. These experiments provide useful insights into the difficulties of candidate discovery, evidence retrieval, and factual comparison.

Critical weaknesses
  1. Insufficient validation of benchmark reliability. WorldFact-Bench introduces factual conflicts through local image editing, but the paper does not sufficiently demonstrate that the edited images are free from detectable manipulation artifacts. Consequently, it remains unclear whether models are identifying factual inconsistencies or exploiting editing-related visual cues. Additional artifact-only baselines or blinded human evaluations would help validate the benchmark.

  2. Limited methodological novelty and insufficient isolation of contributions. PERSIST-Agent combines structured state maintenance, iterative verification, and evidence retrieval, which are closely related to existing agent-based frameworks. Although the ablation results demonstrate the usefulness of individual components, the paper lacks comparisons with simpler iterative agents under equivalent inference budgets and does not isolate the contribution of harness self-optimization.

  3. Limited absolute performance. Despite improvements over the same-backbone baselines, PERSIST-Agent achieves only 18.46% pair accuracy on Qwen3-VL-8B, with 73.71% of pairs receiving identical predictions. This is also below the 25% expected pair accuracy of independent balanced random guessing. These results raise concerns about the practical effectiveness of the proposed verification framework.

  4. Insufficient experimental analysis and reproducibility details. Although the paper specifies shared retrieval limits, it does not adequately compare actual inference costs, including model calls, token consumption, and latency. Moreover, the reported results are averaged over three runs without standard deviations or confidence intervals, while the implementation details of harness self-optimization remain limited.

  5. Limited transparency of dataset annotation quality. While the paper describes several filtering and quality-control stages, it does not clearly report the number of independent annotators, inter-annotator agreement, or quantitative measures of factual label reliability. Given the knowledge-intensive nature of the benchmark, stronger evidence of annotation consistency would improve confidence in the dataset.

Questions for authors
  1. Have the authors evaluated whether real and edited images can be distinguished using manipulation artifacts alone? Could an artifact-focused control experiment or blinded human study be provided to quantify this potential confounding factor?

  2. How much of PERSIST-Agent's improvement comes specifically from the persistent verification state, as opposed to iterative reasoning or harness self-optimization? Could the authors provide a comparison with a simpler iterative agent under the same inference budget, together with an ablation of harness self-optimization?

  3. How should the relatively low absolute pair accuracy be interpreted? Could the authors report uncertainty estimates across repeated runs, together with inference cost comparisons against the retrieval-augmented baselines?

  4. How many human annotators participated in factual verification and quality control? Were annotations independently verified, and is there any measurement of inter-annotator agreement or factual label reliability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer QuN5 (Part 1/2)

We thank the reviewer for recognizing the task definition, source-aligned pair design, and diagnostic experiments. We address your concerns about benchmark reliability, the contribution of the state table, and the interpretation of the results below.

Q1. Could models distinguish the images using editing artifacts alone?

Our evaluation already includes detectors that use generation artifacts. We further examine this factor through quality control and a reverse-editing experiment.

The main comparison includes several relevant approaches:

  • AIDE combines semantic features with low-level signals such as noise patterns.
  • CO-SPY combines semantic and artifact features.
  • FakeVLM is trained for synthetic-image detection and artifact explanation.

Thus, the evaluation covers multiple ways of detecting image-internal clues, rather than only testing general-purpose multimodal models.

Reducing obvious visual defects is also a central construction goal. We seek images that look plausible but contain a specific conflict with real-world facts. Reviewers check whether the intended edit is present, whether non-target content is preserved, and whether the result maintains local structure, visual quality, and target readability. The process is detailed in Appendix A.3–A.4.

The reverse-editing control compares:

  • A: The factually correct original.
  • B: An edited image containing a controlled factual conflict.
  • C: A reverse-edited version of B that restores the correct target fact.

A and C share factual correctness, while B and C both have an editing history.

Appendix A.5, Table 8 covers all 19 models/configurations. For 17 configurations, the absolute accuracy difference between A and C is at most 2.07 percentage points. For the two 8B PERSIST-Agent configurations, C−A is +1.04 and +0.97 points. We also report BC and ABC joint accuracy to distinguish factual discrimination from simply favoring Real.

These results do not show a general tendency to reject images merely because they have been edited. Together with the detector evaluations and quality checks, they support the goal of reducing obvious editing shortcuts and focusing the evaluation on factual discrimination.

Q2. Do the gains come from persistent state, iterative reasoning, or harness optimization?

We added a comparison that separates state representation from prompt optimization. The full-history baseline retains multi-round reasoning and receives an independent optimization run under the same budget.

It retains all previous observations, model outputs, queries, and tool results. Only the separately maintained state table is removed. Both configurations use the same backbone, tools, validation set, optimization budget, and per-image inference limits.

BackboneOptimized full-history agentOptimized state-table agentPair gain
Qwen3-VL-8B16.12%18.46%+2.34 points
InternVL3.5-8B14.96%18.27%+3.31 points

Before optimization, the state-table gains are 1.83 and 3.34 points, respectively. Appendix D.3, Table 16 reports all four settings. The optimization procedure and prompts are provided in Appendix C.3–C.4.

These comparisons show contributions from both state representation and prompt optimization, rather than attributing the total gain to one component.

AuthorsOpenReview ↗

Response to Reviewer QuN5 (Part 2/2)

Q3. How should low Pair accuracy, 73.71% identical predictions, and the 25% random reference be interpreted?

The results show improved pair discrimination over the same-backbone baselines, while reliable factual verification remains unresolved. The random reference and the reported gains describe different aspects of this result.

The 25% expectation applies to a predictor that assigns each image a label independently with equal probability. It is a valid reference, but a model that reasons from image content and learned knowledge is not guaranteed to exceed it.

Our paired images share most visual content and differ in a controlled target fact. Although each image is evaluated separately, a model may apply the same mistaken belief or overlook the same attribute in both. Its prediction errors can therefore be related. The 73.71% identical-label rate shows that the model often fails to distinguish the factual change.

This is why we report F, R, and Pair together. A model can score highly on one class by favoring its label. Pair requires correct predictions on both the original and its edited counterpart, making this failure visible.

On Qwen3-VL-8B, Pair increases from 8.06% for direct judgment and 14.82% for retrieval augmentation to 18.46% for PERSIST-Agent. These comparisons show an improvement, while the score below 25% makes clear that absolute performance remains limited. Appendix B.3 provides the metric definitions and pair-outcome analysis.

WorldFact-Bench is designed to expose this capability gap: finding and verifying factual conflicts from an image without a supplied claim or verification target. PERSIST-Agent demonstrates initial progress, rather than a complete solution.

Q4. Run-to-run variation and inference costs

We provide the three-run mean and sample standard deviation for the two 8B backbones and their main baselines below. Accuracy is reported in percent, and SD is in percentage points.

BackboneMethodFRPair
Qwen3-VL-8BDirect18.51 ± 0.0986.28 ± 0.278.06 ± 1.02
Qwen3-VL-8BRetrieval24.74 ± 0.3876.72 ± 0.4014.82 ± 0.05
Qwen3-VL-8BPERSIST-Agent48.84 ± 0.0661.79 ± 0.5018.46 ± 0.20
InternVL3.5-8BDirect62.94 ± 0.0437.31 ± 0.258.86 ± 0.41
InternVL3.5-8BRetrieval52.97 ± 0.1042.50 ± 0.0410.06 ± 0.05
InternVL3.5-8BPERSIST-Agent41.98 ± 0.0961.67 ± 0.6418.27 ± 0.05

PERSIST-Agent’s Pair standard deviations are 0.20 and 0.05 percentage points, respectively, indicating limited variation in aggregate Pair accuracy across these three runs.

Appendix D.6, Table 19 reports model calls, search calls, and input/output tokens. We additionally provide the mean end-to-end latency per image below.

BackboneMethodMean latency (s/image)
Qwen3-VL-8BDirect6
Qwen3-VL-8BRetrieval24
Qwen3-VL-8BOptimized full-history agent45
Qwen3-VL-8BPERSIST-Agent48
InternVL3.5-8BDirect7
InternVL3.5-8BRetrieval27
InternVL3.5-8BOptimized full-history agent52
InternVL3.5-8BPERSIST-Agent56

Compared with the independently optimized full-history agent, PERSIST-Agent improves Pair accuracy by 2.34/3.31 percentage points, with 6.07%/5.15% more total tokens. Its mean latency is slightly higher: 48 vs. 45 seconds on Qwen3-VL-8B and 56 vs. 52 seconds on InternVL3.5-8B. These results make the accuracy–cost trade-off explicit.

Q5. How many reviewers participated, and how was agreement assessed?

Three reviewers independently assess each candidate at every filtering stage. A candidate proceeds only if all three approve it.

Appendix A.4, Table 7 reports retained and excluded counts, together with the numbers rejected by exactly three, two, or one reviewer. Full agreement ranges from 95.18% to 97.79%, covering all candidates reviewed at each stage, including rejected candidates.

This measures agreement on the stage-specific admission criteria. Factual evidence checks, exclusion of valid variants, and edited-output review jointly support label reliability. The corresponding steps and criteria are detailed in Appendix A.3–A.4.

Reviewer_G6TVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a key blind spot in current research: existing methods primarily focus on detecting generation artifacts or assessing the physical, structural, and semantic plausibility of an image. However, an image can be entirely plausible from a visual standpoint while still conflicting with facts about the physical world. To address this, the study introduces a new benchmark dataset, WorldFact-Bench, designed to evaluate a model's ability to identify discrepancies between an image and external real-world facts. Furthermore, the paper proposes PERSIST-Agent, which maintains a "Persistent Verification State"—a record of candidate facts, visual observations, external evidence, and verification status—to orchestrate subsequent image inspections and information retrieval. The agent’s framework also incorporates a self-evolution mechanism driven by a validation set. Experimental results demonstrate that PERSIST-Agent outperforms both direct-inference and retrieval-augmented baselines when implemented with Qwen3-VL-8B and InternVL3.5-8B.

Critical strengths
  1. The problem perspective is novel: even if an image lacks clear signs of manipulation or obvious semantic errors, it may still contain content that conflicts with factual reality. WorldFact-Bench addresses a gap in this area.
  2. The agent design is ingenious; it employs a "Persistent Verification State" that enables the agent to independently search for evidence, thereby enhancing interpretability. Furthermore, the integration of a self-evolving harness leads to improved performance.
  3. Richly illustrated with text and images.
Critical weaknesses
  1. The paper does not precisely specify the source of the data or the detailed processes involved in its creation.
  2. A broader range of evaluations could be conducted—for instance, by assessing the approach on open-source models of various sizes, such as 30B or 70B parameter models.
  3. There is a lack of detailed information regarding the "self-evolving" harness process.
  4. The accuracy for the two 8B backbone models is approximately 18%, with a peak reported result of 32.20%, indicating that their actual capabilities remain limited.
Questions for authors

I suggest including specific details or concrete examples in the supplementary material addressing the first, second, and third points I raised in the "Key Weaknesses" section; this would help readers better understand the paper. Some of the color combinations in Figure 1, such as those for "Grounded" and "Knowledge Intensive," make the text difficult to read.

2 comments
AuthorsOpenReview ↗

Response to Reviewer G6TV (Part 1/2)

We thank the reviewer for recognizing the value of the task, the persistent verification state, and the presentation.

Our main goal is to address an evaluation gap: can a model identify and verify conflicts between an image and real-world facts without being given a claim, a verification target, or reference evidence? We have added construction details, evaluations on larger backbones, and optimization details in response to your suggestions.

Q1. Data sources and construction details

We have expanded Appendix A.3 to describe source collection, fact verification, local edit design, edited-image review, deduplication, and data splitting.

Each candidate has records of its image source, target fact, and supporting evidence. Each edit changes one factual attribute or relation. Reviewers check that the change creates a factual conflict, does not represent a valid variant, and preserves non-target content.

From 5,600 initial candidates, 1,274 pairs were retained. Appendix A.4, Table 7 provides the filtering counts, review criteria, and agreement statistics. Three reviewers independently assess candidates at every stage, and all three must approve a candidate for it to proceed.

The dataset release will include per-sample sources, target facts, reference evidence, edit instructions, categories, and split assignments.

Q2. Evaluation on models of different sizes

We have expanded the comparison to seven backbone configurations, covering Qwen3-VL 4B, 8B, 32B, and 235B-A22B, and InternVL3.5 4B, 8B, and 38B. Each backbone is evaluated with direct judgment, retrieval augmentation, and PERSIST-Agent.

As reported in Appendix D.4, Table 17, PERSIST-Agent achieves:

BackbonePair accuracy
Qwen3-VL-32B29.83%
InternVL3.5-38B31.46%
Qwen3-VL-235B-A22B43.27%

Across all seven backbones, PERSIST-Agent improves Pair by 8.51–14.09 percentage points over direct judgment and 3.64–8.21 points over retrieval augmentation. These results show that the gains extend beyond the two 8B backbones.

AuthorsOpenReview ↗

Response to Reviewer G6TV (Part 2/2)

Q3. Details of the self-evolving harness

Appendix C.3, Table 14 now specifies the optimizer model, validation set, optimization rounds, candidate count, trace feedback, and selection procedure.

Each configuration uses 274 validation pairs, with 4 rounds and 3 new candidates per round. Including the initial prompt, this gives 13 full validation evaluations. The optimizer receives previous prompts, their F/R/Pair scores, and sampled successful and failed traces. Model weights and tool interfaces remain fixed, and the selected prompts are frozen before testing.

Appendix C.4 presents the initial and optimized instructions for both agent configurations, plus the optimizer instructions. Appendix D.3, Table 16 compares agents with and without the state table, before and after optimization, to separate the two contributions.

Q4. Limited absolute performance

We agree that absolute performance remains limited. Our goal is to establish an evaluation setting for this capability and demonstrate initial progress toward solving it.

Detecting editing artifacts, checking image–text agreement, and verifying image content against real-world facts address different questions. An image can look natural and match a description while still containing a factual error.

WorldFact-Bench focuses on this gap. Given only the current image, a model must identify what to verify, obtain or recall relevant knowledge, and determine whether a factual conflict exists. Source-aligned pairs further test whether the model responds correctly to a controlled factual change.

We support this evaluation setting with controlled edits, factual evidence, human review, and the reverse-editing control in Appendix A.5.

PERSIST-Agent improves over the same-backbone baselines, and the larger-backbone experiments reach 43.27% Pair accuracy. Reliable verification nevertheless remains an open problem. The failure analysis in Appendix D.5, Table 18 identifies target discovery/visual understanding and evidence reasoning as the main remaining difficulties. We hope WorldFact-Bench helps future research identify and improve these capabilities.

Q5. Text readability in Figure 1

We have darkened the text in Figure 1, including the “Grounded” and “Knowledge Intensive” labels, to improve contrast with the background.

Reviewer_ivENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

his paper introduces WorldFact-Bench, containing 1,274 source-aligned original–edited image pairs for detecting factual inconsistencies without a predefined verification target. It also proposes PERSIST-Agent, which maintains candidate-level records of visual observations, retrieved evidence, and verification status. The paper reports improvements over direct and retrieval-augmented judgment on two 8B backbones, supported by component ablations and diagnostic analyses.

Critical strengths
  • The benchmark addresses a useful distinction between visual plausibility and factual consistency. Source-aligned pairs and independent-image evaluation provide a focused test of factual discrimination.
  • The same-backbone comparisons, oracle experiments, and failure analyses offer useful diagnostic coverage beyond aggregate accuracy.
Critical weaknesses
  • Benchmark validity needs stronger validation. Although the paper describes human review and exclusion of valid variants, it does not quantify annotation agreement or provide edits that preserve factual correctness as controls. These checks are important for distinguishing factual-verification failures from ambiguous labels and sensitivity to editing artifacts.
  • The specific benefit of persistent state is insufficiently isolated. Shared search limits do not establish matched iterative computation or prompt tuning. The paper lacks a full-history iterative baseline with comparable optimization and an ablation without harness optimization. Appendix C also omits the optimizer identity, optimization budget, and final prompts, limiting reproducibility.
Questions for authors
  • Can the authors report independent annotation agreement and evaluate fact-preserving edits to distinguish factual inconsistency from image manipulation itself?
  • Does the state-table advantage persist against a full-history iterative baseline with matched inference and prompt-optimization budgets?
  • What are the results without harness optimization, and can the optimizer configuration and final prompts be provided?
  • How should the reported gains be interpreted relative to the 25% expected Pair accuracy of independent fair random predictions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ivEN (Part 1/2)

We thank the reviewer for recognizing the value of the task, the source-aligned pairs, and the diagnostic experiments. We have added validation experiments and implementation details to address your concerns.

Q1. Annotation agreement and fact-preserving edits

We have added both human-review statistics and a reverse-editing control.

Three reviewers independently assessed each candidate at every filtering stage. A candidate was retained only if all three approved it. Appendix A.4, Table 7 reports retained and excluded counts, including candidates rejected by exactly three, two, or one reviewer. Full three-reviewer agreement ranges from 95.18% to 97.79%, measured over all candidates at each stage, including rejected ones.

For the editing control, we compare:

  • A: The factually correct original.
  • B: Its edited version containing a factual conflict.
  • C: A reverse-edited version of B that restores the correct target fact.

A and C share factual correctness, while B and C both have an editing history. This helps separate factual discrimination from sensitivity to editing.

Appendix A.5, Table 8 covers all 19 models/configurations in the main comparison. For 17 configurations, the absolute accuracy difference between A and C is at most 2.07 percentage points. For the two 8B PERSIST-Agent configurations, C−A is +1.04 and +0.97 points. We also report BC and ABC joint accuracy to distinguish factual discrimination from simply favoring Real.

These results support the view that editing history alone is not the main driver of predictions in this control.

Q2. State-table benefit against an independently optimized full-history baseline

The state-table advantage remains after giving the full-history baseline the same optimization budget.

The baseline retains multi-round search and all previous observations, model outputs, and tool results. Only the separately maintained state table is removed. Both configurations use the same backbone, validation set, optimization budget, and per-image inference limits.

BackboneOptimized full-history agentOptimized state-table agentPair gain
Qwen3-VL-8B16.12%18.46%+2.34 points
InternVL3.5-8B14.96%18.27%+3.31 points

See Appendix D.3, Table 16 for the full comparison and Appendix D.6, Table 19 for model calls and token use.

The original w/o State ablation directly removed state-table instructions from the optimized method without further optimization. The new baseline is independently optimized, directly addressing your concern about comparable optimization opportunities.

AuthorsOpenReview ↗

Response to Reviewer ivEN (Part 2/2)

Q3. Results without harness optimization, optimizer settings, and prompts

Without harness optimization, the state-table agent achieves 14.21% and 15.08% Pair accuracy on the two 8B backbones, compared with 18.46% and 18.27% after optimization. Appendix D.3, Table 16 reports both agent configurations before and after optimization.

Appendix C.3, Table 14 provides the optimizer model and settings: 274 validation pairs, 4 rounds, and 3 new candidates per round, giving 13 full validation evaluations including the initial prompt.

Feedback includes previous prompts, measured F/R/Pair scores, and sampled successful and failed traces. Selection focuses primarily on Pair, with F/R and traces used to examine label bias and the source of improvements. Prompts are frozen before testing.

Appendix C.4 presents five complete sets of instructions: initial and optimized prompts for both agent configurations, plus the RSI optimizer instructions.

Q4. Interpretation relative to the 25% independent fair-random reference

The 25% expectation is a valid reference for independent, equally likely random labels. Our reported gains measure improvement over the same-backbone baselines, while the score below this reference highlights the remaining difficulty of the task.

The paired images share most visual content and differ in a controlled target fact. Although they are evaluated separately, a model may apply the same mistaken belief, overlook the same attribute, or favor the same label on both images. Its prediction errors can therefore be related, and reasoning from image content does not guarantee a Pair score above 25%.

This is why we report F, R, and Pair together. Always predicting Real gives 100% R but zero F and Pair. Pair requires correct predictions on both the original and its edited counterpart, exposing failures that a high score on one class can hide.

For Qwen3-VL-8B, PERSIST-Agent raises Pair from 8.06% to 18.46%, gaining 10.40 percentage points over direct judgment and 3.64 points over the retrieval-augmented baseline. These results show improved factual discrimination. Nevertheless, 73.71% of pairs still receive identical labels, and Pair remains below the 25% random reference. Reliable verification therefore remains unresolved.

The benchmark reveals this capability gap, while the same-backbone comparisons measure progress toward addressing it. Appendix B.3 now clarifies the random reference and reports the pair-outcome analysis alongside F/R/Pair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.