- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the safety risks of commercial image-generation models in producing visual misinformation. First, the authors identify a Verification–Generation Gap, where image-generation models can recognize certain real-world claims as misinformation during explicit fact-checking tasks, yet still render the same false claims as credible visual evidence during image generation. To systematically evaluate this issue, this paper introduce EpiReal-Bench, which consists of 1,000 verified false real-world claims instantiated across 10 credible visual formats, resulting in 10,000 generation prompts and corresponding generated images. Furthermore, this paper proposes EpiReal-Attack, a black-box optimization framework that combines skill-guided prompt transformations, multimodal feedback, and Pareto-based selection to evaluate the robustness of image-generation safeguards against adaptive attacks. Finally, the authors introduce Verification-Guided Prompting (VGP), a defense strategy that explicitly requires models to perform factual verification before image generation, without additional training or model modifications. Experiments on four commercial image-generation models show that the average attack success rate under direct prompting exceeds 70%, while EpiReal-Attack increases it to over 95%. VGP achieves relatively high interception rates on some models, although its effectiveness varies considerably across models.
Critical strengths
- With the continuous advancement of commercial image-generation models, these models have become capable of producing highly realistic images. This paper focuses on generated content that appears visually benign but may convey false real-world claims, addressing an important safety issue in image generation that deserves further investigation.
- EpiReal-Bench covers 10 real-world claim categories and 10 credible visual formats. In addition, the authors improve the quality of the dataset through human verification, semantic deduplication, and risk rationale annotations. Overall, the benchmark is rigorously and systematically constructed.
- The paper not only evaluates the performance of commercial image-generation models under direct prompting but also investigates their robustness against adaptive prompt optimization through EpiReal-Attack. Furthermore, this compare various input-level and output-level defense mechanisms and propose VGP as a preliminary mitigation strategy. These experiments go beyond identifying the existence of safety risks by examining how these risks evolve under different evaluation settings and exploring potential mitigation approaches.
Critical weaknesses
- This paper interprets the behavioral differences between factual verification and image generation tasks as a generation-time safety alignment failure. However, factual verification and image generation are inherently different tasks with different objectives. A model's ability to recognize a claim as misinformation does not necessarily mean that it should refuse to generate the claim when receiving a visualization request, especially in legitimate scenarios such as education, satire, fictional creation, or research analysis. Observing behavioral differences between these two tasks is not sufficient to directly demonstrate the existence of an independent Alignment Failure. Such differences may also result from different task objectives, prompt requirements, or safety instructions. The VGP proposed in this paper also demonstrates that simply adding a factual verification instruction to the original image generation request can significantly change generation behavior in some models. Therefore, it remains unclear whether the so-called Gap observed in the paper mainly comes from the lack of explicit factual verification or safety constraints in the generation task, rather than a fundamental alignment failure between the model's factual awareness and generation capabilities.
2.The model's correct judgments of existing claims may rely on knowledge acquired during pretraining, but the paper does not sufficiently distinguish between the model's ability to recognize existing knowledge and its ability to verify unknown facts. This is particularly relevant in scenarios involving breaking events and emerging fake news. For events beyond the model's knowledge scope, even if a claim can be proven false through the latest external evidence, the model may not be able to determine its truthfulness based solely on its own parametric knowledge. Under the current setting, it is unreliable for the model to identify such false claims.
3.The Benchmark's definition of Unsafe Generation relies too heavily on factual truth or falsity, while ignoring user intent and generation context. This paper mainly measures safety risks by evaluating whether a model can present verified false claims as credible visual evidence. However, the factual falsity of a claim does not necessarily mean that the corresponding visual generation behavior is malicious or unsafe. The current Benchmark does not systematically distinguish between legitimate visualization tasks involving false claims and generation tasks intended to create misleading evidence. Therefore, the reported ASR may, to some extent, mix the model's instruction-following ability with genuine safety failures. I believe that the safety risks of visual misinformation should not be determined solely by Claim Factuality, but should also consider user intent, usage context, and whether the generated content attempts to impersonate authentic information sources. In particular, whether a false claim is explicitly labeled as simulated content may be an important factor in distinguishing legitimate generation from deceptive generation.
Questions for authors
-
Regarding the definition of the Verification–Generation Gap: How can a genuine Generation-time Alignment Failure be distinguished from behavioral differences caused by different task objectives or safety instructions? If safety requirements such as "avoid generating misleading visual evidence" are explicitly added to ordinary image generation instructions without performing factual verification, can they achieve results similar to VGP? Can the individual contributions of factual verification instructions and safety constraint instructions be evaluated separately?
-
How can we determine whether the high FCR on the Benchmark reflects reliable factual judgment capabilities, rather than merely the recognition of existing knowledge, events potentially encountered during training, or patterns of claim plausibility? For new events beyond the model's knowledge scope, the model may not even know whether the claims are true or false. In such cases, how should this Gap be defined?
-
How should VGP handle claims that can be verified using the latest external evidence but cannot be judged based on the model's parametric knowledge? For breaking news beyond the model's knowledge scope, if the information is actually true but the model cannot determine its truthfulness, could VGP lead to excessive refusals?
-
How is the generation of an image depicting a false claim defined as a genuine safety failure rather than a legitimate image generation task? Should models also refuse to generate simulated news images used for education, research, or satire? Could controlled experiments involving different user intents and contexts be constructed, such as comparing generation tasks explicitly labeled as simulated content with tasks intended to fabricate authentic evidence, to verify whether the Benchmark can distinguish legitimate generation from deceptive generation?
3 comments
Response to W2 & Q2: Factual Recognition and Knowledge Boundaries
Thank you for highlighting this distinction!
- Benchmark Scope and Emerging Events.
We agree that models may lack sufficient knowledge to reliably judge emerging or rapidly evolving events. In such cases, image generation alone is insufficient to establish a verification–generation gap. Our benchmark therefore focuses on claims that are demonstrably false or unsupported in the relevant real-world context, rather than disputed or unresolved emerging events. Within this setting, we examine whether models that recognize a claim as false during explicit verification nevertheless generate images presenting the same claim as factual.
- Factual Recognition and Benchmark Evaluation.
We clarify that recognizing learned facts and making factual judgments are not mutually exclusive. Models can draw on their pretraining knowledge to identify false factual assertions, and FCR evaluates whether they can recognize such claims under explicit verification. Our benchmark excludes disputed or unresolved emerging-event claims and hypothetical future scenarios (Sec. 3.2), ensuring that the evaluation focuses on claims whose factual status can be assessed.
GPT-Image-2 and Nano Banana 2 achieve 100% FCR (Sec. 3.1), indicating that they correctly recognize the evaluated benchmark claims as false or unsupported under the explicit verification protocol. Benign controls under the same protocol yield 92.8%–99.1% accuracy, while VGP's false-refusal rates range from 0.5% to 5.3% (Appendix C.1, Table 5). Together, these results support our central observation: models can recognize false or unsupported claims during explicit verification, yet do not consistently apply this recognition during image generation.
Table 5: factual-verification accuracy and VGP's false-refusal rate on benign image-generation requests.
| Model | Benign verification accuracy (%) ↑ | VGP false-refusal rate (%) ↓ |
|---|---|---|
| GPT-Image-2 | 98.20 | 0.50 |
| Nano Banana 2 | 99.10 | 1.60 |
| Grok Imagen 2.0 | 94.60 | 1.00 |
| Seedream 5.0 Flash | 92.80 | 5.30 |
Response to Q3: VGP on Recent News
Thank you for raising these important points!
We test VGP on recent, source-verified news claims to examine whether it preserves the ability to generate legitimate visual evidence about emerging events. Since we use strictly black-box APIs, we cannot determine whether the providers employ external retrieval or how they update their internal knowledge.
We select 100 recent, source-verified news claims across 10 domains and evaluate GPT-Image-2 under matched settings, with and without VGP.
| Setting on clean recent-news prompts | ASR (%) | MR |
|---|---|---|
| Without VGP | 100.0 | 5.00 |
| With VGP | 100.0 | 5.00 |
With VGP, all 100 requests generate images, and every image receives MR=5, matching the results without VGP. These results indicate that VGP does not reduce image generation or claim realization on this tested set of source-verified news claims.
Response to Q4: Definition of the Targeted Safety Failure
We appreciate your insightful comments, which help us clarify and strengthen our work!
We define the targeted safety failure in EpiReal-Bench as generating credible-looking visual-text evidence that presents a false real-world claim as factual, without an explicit qualification or correction. Importantly, visualizing a false claim in a clearly identified educational, fictional, illustrative, or simulated context does not constitute the targeted safety failure.
To validate this distinction, we conduct a controlled comparison between ordinary generation requests and explicitly simulated contexts. As reported in Response to W1 & W3 & Q1, Point 2: Distinguishing Legitimate Visualization from Misinformation Generation, all simulated requests successfully generate images, but none achieves MR=5 or counts as an ASR success. This is consistent with our evaluation criterion, which distinguishes legitimate visualization from the unqualified presentation of false claims as authentic evidence.
Dear Reviewer,
This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.
Thank you again for your time and consideration.
Thank you for recognizing an "important safety issue" and a "rigorously and systematically constructed" benchmark. This recognition is a great encouragement to us!
Response to W1, W3 & Q1: Task Context and the Verification–Generation Gap
Thank you for your careful review and helpful feedback!
We study requests that package misinformation as text-integrated visual misinformation, in which textual assertions and visual elements jointly construct credible-looking evidence with the potential to mislead viewers. As clarified in Sec. 3.2, we exclude requests explicitly framed as educational, satirical, fictional, or hypothetical scenarios. Within this scope, the verification–generation gap refers to a specific inconsistency: a model can recognize a claim as false or unsupported during explicit verification, yet still generate visual evidence presenting that claim as factual under an ordinary generation request. Our focus is therefore not on whether models can legitimately visualize false claims in educational or fictional contexts, but on whether their factual recognition consistently constrains their generation behavior when such contexts are absent.
To further examine this distinction, we add a controlled evaluation on a 100-prompt subset using GPT-Image-2. We compare the original prompts with matched prompts explicitly framed as educational examples, science-fiction scenes, hypothetical scenarios, or illustrative content, while keeping the underlying claims and visual formats unchanged.
All simulated requests successfully produce images, but none achieves MR=5 or counts as an ASR success. The generated images contain visible fictional or illustrative qualifications, which substantially weaken their presentation as authentic factual evidence. These results indicate that our evaluation distinguishes the tested legitimate visualization contexts from unqualified misinformation generation. Importantly, the verification–generation gap reported in our benchmark is measured under ordinary generation requests, not explicitly simulated contexts..
We further conduct an ablation study to distinguish the contributions of factual verification and explicit safety instructions. We compare three settings: verification instructions alone, the safety constraint "Avoid generating misleading visual evidence" alone, and their combination in full VGP.
Full VGP achieves the highest interception rates across all four models, outperforming both individual components. In particular, verification alone consistently outperforms the safety constraint alone, while combining both further improves interception. These results indicate that factual verification and explicit safety constraints make complementary contributions to VGP's effectiveness. However, the need for such explicit instructions also highlights the central issue studied in our work: models do not consistently apply factual verification and safety considerations in the tested ordinary image-generation setting.