- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper points out that whether an image is AI-generated does not necessarily indicate whether the situation it depicts is true or false. To address this, the authors introduce the task of "Image Factuality Assessment." In this task, a system is provided only with an image and a general verification prompt, without any accompanying textual claims, and must determine whether the depicted situation is "substantiated" or "refuted," while providing an explanation based on external evidence. For this task, the authors developed a benchmark called EviLens and proposed a multimodal agent named EviScene. EviScene operates by iteratively performing visual inspection, external information retrieval, and evidence comparison, progressively refining the questions under investigation. The authors trained the Qwen3.5-9B model using a two-stage process, enabling it to achieve a balanced accuracy and a strict evidence-sufficiency rate.
Critical strengths
The authors collected and manually reviewed a large volume of data for EviLens. The authors conducted extensive testing of EviLens using a variety of both open-source and closed-source models.
Critical weaknesses
- Although the authors have collected and organized a large dataset, the distribution is quite imbalanced (3,114 substantiated images versus 7,060 refuted images). I believe the dataset could be smaller, provided that a better balance is maintained.
- The authors do not compare how different agent architectures affect task completion.
- Some experimental results in the paper lack analysis, making them difficult to interpret.
Questions for authors
- Could the authors clarify the distinction between the "Image Factuality Assessment" proposed in this paper and general multimodal fact-checking tasks?
- Could the authors clarify the final conclusions drawn in Sections 5.4 and 5.5? Currently, these sections merely present experimental results without providing sufficient analysis.
- Table 1 shows a significant disparity in SESR scores between models that do not employ an agent and those that do, while the differences across other experimental metrics remain relatively small. Could the authors provide a reasonable explanation for this anomaly?
3 comments
Revised Sections 5.4 and 5.5
5.4 Source and Generator Analysis. The gains from training extend to web-collected images and to generators unseen in training (Table 2). The full policy performs similarly on constructed and web-collected images, with 76.75% versus 77.21% BAcc and 52.75% versus 50.17% SESR. Training improves both sources: relative to the base agent, Full gains 8.88 BAcc and 8.81 SESR points on constructed images, and 3.87 BAcc and 11.76 SESR points on web-collected images. Across generators, Full exceeds the base agent on images from all three, including GPT Image 2 and Seedream 5.0 Lite, which are not used to construct training images.
5.5 Qualitative Analysis. Prior knowledge and visual plausibility cannot substitute for evidence about the depicted situation. Figure 5 shows two complementary failure patterns of direct judgment by GPT-5.5. The first pattern treats missing knowledge as evidence of falsity. In Figure 5a, the display label names a DisplayHDR True Black 1400 certification, and GPT-5.5 rejects it because it does not know this tier; EviScene finds that VESA introduced the tier in July 2026. In Figure 5d, the image shows the launch of Plan Cabaret at the Moulin Rouge, and GPT-5.5 calls the program invented; EviScene finds the Ministry of Culture record of the launch on January 21, 2025. The second pattern accepts a plausible scene without checking the decisive detail. In Figure 5b, the Cassini poster names the correct exhibition, institution, and venue, so GPT-5.5 accepts it; EviScene retrieves the official schedule, which runs from October 14, 2026 to February 1, 2027, whereas the poster shows 2025 to 2026. In Figure 5c, GPT-5.5 accepts the depicted opening ceremony; EviScene finds official records placing berlin modern in its construction stage as of the evaluation reference date of September 1, 2026. In all four cases, GPT-5.5 knows the current date and the entities involved but lacks the specific record that decides the verdict.
Dear Reviewer 2KqQ,
Thank you again for raising the questions of agent architecture and label imbalance. In our response, we compared EviScene with variants that fix the verification question, the action sequence, or both; fixing them lowers BAcc by 3.25 to 5.74 points and SESR by 3.20 to 6.09 points (Table 4a). The imbalance does not bias the trained policy toward the majority class: it recalls 80.11% of supported and 78.00% of refuted images (Table 1). We also rewrote Sections 5.4 and 5.5 to state their conclusions. If you have any further questions, or if any part of our response is unclear, we would be glad to continue the discussion.
Best regards, The Authors
We thank the reviewer for the comments and address each point below. We have uploaded a revised PDF with changes in blue; all references below are to it.
The imbalance does not affect our results. BAcc and macro F1 weight both classes equally, so labeling every image refuted yields 50% BAcc. The trained policy recalls 80.11% of supported and 78.00% of refuted images (Table 1). On web-collected images, which are close to balanced with 144 supported and 145 refuted test images (Table 8), it reaches 77.21% BAcc, compared with 76.75% on the imbalanced constructed images (Table 2). The imbalance comes from controlled construction, which builds 502 to 524 refuted images for each of twelve error types so that each type is well represented (Table 9).
We add three variants on the full test set to Section 5.3 and Table 4a, all using Gemini 3.7 Flash with the same tools and 24-action budget. With a fixed question, the agent states one verification question from the image before any tool call and keeps it throughout. With a fixed sequence, every image goes through the same nine actions: scene perception, OCR, reverse image search, three text searches, and three page visits; the agent only fills in the tool arguments.
EviScene, which adapts both, reaches the highest BAcc and SESR with the fewest actions. Fixing the question, the sequence, or both lowers BAcc by 4.56, 3.25, and 5.74 points and SESR by 4.32, 3.20, and 6.09 points; fixing the question also nearly doubles the actions per image. A question fixed before any evidence commits the investigation to one detail, whereas EviScene revises what it checks as findings accumulate.
We rewrote both sections to state their conclusions; the revised text is in the PDF and the next comment. Section 5.4 concludes that the gains from training extend to web-collected images and to generators unseen in training. Section 5.5 concludes that prior knowledge and visual plausibility cannot substitute for evidence about the depicted situation.
Multimodal fact-checking depends on a given claim, which states what is to be verified. In MOCHEG (Yao et al., 2023), the claim is text and images appear among the retrieved evidence. In AVerImaTeC (Cao et al., 2025), RW-Post (Xu et al., 2026), and VeriTaS (Rothermel et al., 2026), a textual claim is paired with images or with the social-media post that carried it. When a claim is only implicit in an image, the standard pipeline first extracts it in a separate claim-extraction stage, so verification still starts from a given claim. Image Factuality Assessment supplies no claim and has no extraction stage: the system receives one image and a generic instruction, decides during investigation which depicted facts to check, and judges the depicted situation as a whole.
Table 5 in Appendix A.1 further compares EviLens with benchmarks for related tasks.
This gap is expected, and Section 5.2 now explains it. BAcc checks only the label, whereas SESR also requires the response to establish the decisive fact with evidence. Without investigation, a model has no such evidence, and its labels can follow a default preference: under direct judgment, Gemini 3.1 Pro recalls 96.52% of refuted but only 28.65% of supported images, while GLM-4.6V shows the reverse (Table 1). Such a label can therefore match the ground truth while its grounds do not establish the decisive fact, which keeps SESR near zero. Within investigation, SESR likewise follows how much evidence the agent can obtain: raising the action limit of the full policy from 8 to 16 adds 12.05 SESR points but only 0.79 BAcc points (Table 4b), and removing web search lowers SESR by 16.37 points but BAcc by only 4.68 (Table 3). The gap therefore measures the evidence that direct judgment cannot obtain.