acceptodds
Withdrawn from ICLR 2027

False Claims, Credible Images: A Red-Teaming Benchmark for Commercial Image Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image-generation models can now produce text-rich, natural-looking visual artifacts that are hard to distinguish from real-world evidence, such as news reports and textbook pages. Yet the same capability introduces a new risk: these models can just as easily fabricate visual misinformation. Even commercial models (e.g., GPT-Image-2) readily produce it. Curiously, we find that these models can recognize a claim as false when asked, yet still render that very claim as credible visual evidence. This discrepancy points to a blind spot in current alignment: safeguards judge what an image shows, not what it asserts. However, existing red-teaming benchmarks target conventional harmful content, such as violent or explicit imagery, and say little about where the alignment boundaries lie for visual misinformation, especially in commercial models. To fill this gap, we introduce EpiReal-Bench, the first systematic benchmark for evaluating visual misinformation risks in commercial image generators, comprising false-claim prompts spanning real-world claim categories and credible visual formats. We further introduce EpiReal-Attack, a skill-guided black-box optimization framework that uses Pareto-based selection and multimodal feedback to identify prompts that bypass alignment safeguards while preserving visual realism, textual legibility, and semantic fidelity. Experiments on four commercial models reveal that over of false-claim prompts elicit images that faithfully depict the corresponding misinformation, and EpiReal-Attack pushes this rate to . Perhaps most worrying, these models are only a click away, and their outputs are cheap to spread yet hard to disbelieve, leaving this dimension of alignment largely unguarded.

Primary area:
datasets and benchmarks
Keywords:
Image-generation model, AI Safety, Visual Misinformation, Safety Alignment, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MgWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the safety risks of commercial image-generation models in producing visual misinformation. First, the authors identify a Verification–Generation Gap, where image-generation models can recognize certain real-world claims as misinformation during explicit fact-checking tasks, yet still render the same false claims as credible visual evidence during image generation. To systematically evaluate this issue, this paper introduce EpiReal-Bench, which consists of 1,000 verified false real-world claims instantiated across 10 credible visual formats, resulting in 10,000 generation prompts and corresponding generated images. Furthermore, this paper proposes EpiReal-Attack, a black-box optimization framework that combines skill-guided prompt transformations, multimodal feedback, and Pareto-based selection to evaluate the robustness of image-generation safeguards against adaptive attacks. Finally, the authors introduce Verification-Guided Prompting (VGP), a defense strategy that explicitly requires models to perform factual verification before image generation, without additional training or model modifications. Experiments on four commercial image-generation models show that the average attack success rate under direct prompting exceeds 70%, while EpiReal-Attack increases it to over 95%. VGP achieves relatively high interception rates on some models, although its effectiveness varies considerably across models.

Critical strengths
  1. With the continuous advancement of commercial image-generation models, these models have become capable of producing highly realistic images. This paper focuses on generated content that appears visually benign but may convey false real-world claims, addressing an important safety issue in image generation that deserves further investigation.
  2. EpiReal-Bench covers 10 real-world claim categories and 10 credible visual formats. In addition, the authors improve the quality of the dataset through human verification, semantic deduplication, and risk rationale annotations. Overall, the benchmark is rigorously and systematically constructed.
  3. The paper not only evaluates the performance of commercial image-generation models under direct prompting but also investigates their robustness against adaptive prompt optimization through EpiReal-Attack. Furthermore, this compare various input-level and output-level defense mechanisms and propose VGP as a preliminary mitigation strategy. These experiments go beyond identifying the existence of safety risks by examining how these risks evolve under different evaluation settings and exploring potential mitigation approaches.
Critical weaknesses
  1. This paper interprets the behavioral differences between factual verification and image generation tasks as a generation-time safety alignment failure. However, factual verification and image generation are inherently different tasks with different objectives. A model's ability to recognize a claim as misinformation does not necessarily mean that it should refuse to generate the claim when receiving a visualization request, especially in legitimate scenarios such as education, satire, fictional creation, or research analysis. Observing behavioral differences between these two tasks is not sufficient to directly demonstrate the existence of an independent Alignment Failure. Such differences may also result from different task objectives, prompt requirements, or safety instructions. The VGP proposed in this paper also demonstrates that simply adding a factual verification instruction to the original image generation request can significantly change generation behavior in some models. Therefore, it remains unclear whether the so-called Gap observed in the paper mainly comes from the lack of explicit factual verification or safety constraints in the generation task, rather than a fundamental alignment failure between the model's factual awareness and generation capabilities.

2.The model's correct judgments of existing claims may rely on knowledge acquired during pretraining, but the paper does not sufficiently distinguish between the model's ability to recognize existing knowledge and its ability to verify unknown facts. This is particularly relevant in scenarios involving breaking events and emerging fake news. For events beyond the model's knowledge scope, even if a claim can be proven false through the latest external evidence, the model may not be able to determine its truthfulness based solely on its own parametric knowledge. Under the current setting, it is unreliable for the model to identify such false claims.

3.The Benchmark's definition of Unsafe Generation relies too heavily on factual truth or falsity, while ignoring user intent and generation context. This paper mainly measures safety risks by evaluating whether a model can present verified false claims as credible visual evidence. However, the factual falsity of a claim does not necessarily mean that the corresponding visual generation behavior is malicious or unsafe. The current Benchmark does not systematically distinguish between legitimate visualization tasks involving false claims and generation tasks intended to create misleading evidence. Therefore, the reported ASR may, to some extent, mix the model's instruction-following ability with genuine safety failures. I believe that the safety risks of visual misinformation should not be determined solely by Claim Factuality, but should also consider user intent, usage context, and whether the generated content attempts to impersonate authentic information sources. In particular, whether a false claim is explicitly labeled as simulated content may be an important factor in distinguishing legitimate generation from deceptive generation.

Questions for authors
  1. Regarding the definition of the Verification–Generation Gap: How can a genuine Generation-time Alignment Failure be distinguished from behavioral differences caused by different task objectives or safety instructions? If safety requirements such as "avoid generating misleading visual evidence" are explicitly added to ordinary image generation instructions without performing factual verification, can they achieve results similar to VGP? Can the individual contributions of factual verification instructions and safety constraint instructions be evaluated separately?

  2. How can we determine whether the high FCR on the Benchmark reflects reliable factual judgment capabilities, rather than merely the recognition of existing knowledge, events potentially encountered during training, or patterns of claim plausibility? For new events beyond the model's knowledge scope, the model may not even know whether the claims are true or false. In such cases, how should this Gap be defined?

  3. How should VGP handle claims that can be verified using the latest external evidence but cannot be judged based on the model's parametric knowledge? For breaking news beyond the model's knowledge scope, if the information is actually true but the model cannot determine its truthfulness, could VGP lead to excessive refusals?

  4. How is the generation of an image depicting a false claim defined as a genuine safety failure rather than a legitimate image generation task? Should models also refuse to generate simulated news images used for education, research, or satire? Could controlled experiments involving different user intents and contexts be constructed, such as comparing generation tasks explicitly labeled as simulated content with tasks intended to fabricate authentic evidence, to verify whether the Benchmark can distinguish legitimate generation from deceptive generation?

3 comments

Thank you for recognizing an "important safety issue" and a "rigorously and systematically constructed" benchmark. This recognition is a great encouragement to us!

Response to W1, W3 & Q1: Task Context and the Verification–Generation Gap

Thank you for your careful review and helpful feedback!

  1. Scope of the Task and the Observed Alignment Failure.

We study requests that package misinformation as text-integrated visual misinformation, in which textual assertions and visual elements jointly construct credible-looking evidence with the potential to mislead viewers. As clarified in Sec. 3.2, we exclude requests explicitly framed as educational, satirical, fictional, or hypothetical scenarios. Within this scope, the verification–generation gap refers to a specific inconsistency: a model can recognize a claim as false or unsupported during explicit verification, yet still generate visual evidence presenting that claim as factual under an ordinary generation request. Our focus is therefore not on whether models can legitimately visualize false claims in educational or fictional contexts, but on whether their factual recognition consistently constrains their generation behavior when such contexts are absent.

  1. Distinguishing Legitimate Visualization from Misinformation Generation.

To further examine this distinction, we add a controlled evaluation on a 100-prompt subset using GPT-Image-2. We compare the original prompts with matched prompts explicitly framed as educational examples, science-fiction scenes, hypothetical scenarios, or illustrative content, while keeping the underlying claims and visual formats unchanged.

SettingGeneration Rate (%)ASR (%)MRVC
EpiReal-Bench87744.623.58
Simulated content10001.571.74

All simulated requests successfully produce images, but none achieves MR=5 or counts as an ASR success. The generated images contain visible fictional or illustrative qualifications, which substantially weaken their presentation as authentic factual evidence. These results indicate that our evaluation distinguishes the tested legitimate visualization contexts from unqualified misinformation generation. Importantly, the verification–generation gap reported in our benchmark is measured under ordinary generation requests, not explicitly simulated contexts..

  1. Contributions of Factual Verification and Safety Instructions.

We further conduct an ablation study to distinguish the contributions of factual verification and explicit safety instructions. We compare three settings: verification instructions alone, the safety constraint "Avoid generating misleading visual evidence" alone, and their combination in full VGP.

SettingGPT-Image-2Nano Banana 2Grok Imagen 2.0Seedream 5.0 Flash
Verification only52.461.126.238.5
Safety constraint only40.946.825.729.1
VGP (Full)76.293.035.768.4

Full VGP achieves the highest interception rates across all four models, outperforming both individual components. In particular, verification alone consistently outperforms the safety constraint alone, while combining both further improves interception. These results indicate that factual verification and explicit safety constraints make complementary contributions to VGP's effectiveness. However, the need for such explicit instructions also highlights the central issue studied in our work: models do not consistently apply factual verification and safety considerations in the tested ordinary image-generation setting.

Response to W2 & Q2: Factual Recognition and Knowledge Boundaries

Thank you for highlighting this distinction!

  1. Benchmark Scope and Emerging Events.

We agree that models may lack sufficient knowledge to reliably judge emerging or rapidly evolving events. In such cases, image generation alone is insufficient to establish a verification–generation gap. Our benchmark therefore focuses on claims that are demonstrably false or unsupported in the relevant real-world context, rather than disputed or unresolved emerging events. Within this setting, we examine whether models that recognize a claim as false during explicit verification nevertheless generate images presenting the same claim as factual.

  1. Factual Recognition and Benchmark Evaluation.

We clarify that recognizing learned facts and making factual judgments are not mutually exclusive. Models can draw on their pretraining knowledge to identify false factual assertions, and FCR evaluates whether they can recognize such claims under explicit verification. Our benchmark excludes disputed or unresolved emerging-event claims and hypothetical future scenarios (Sec. 3.2), ensuring that the evaluation focuses on claims whose factual status can be assessed.

GPT-Image-2 and Nano Banana 2 achieve 100% FCR (Sec. 3.1), indicating that they correctly recognize the evaluated benchmark claims as false or unsupported under the explicit verification protocol. Benign controls under the same protocol yield 92.8%–99.1% accuracy, while VGP's false-refusal rates range from 0.5% to 5.3% (Appendix C.1, Table 5). Together, these results support our central observation: models can recognize false or unsupported claims during explicit verification, yet do not consistently apply this recognition during image generation.

Table 5: factual-verification accuracy and VGP's false-refusal rate on benign image-generation requests.

ModelBenign verification accuracy (%) ↑VGP false-refusal rate (%) ↓
GPT-Image-298.200.50
Nano Banana 299.101.60
Grok Imagen 2.094.601.00
Seedream 5.0 Flash92.805.30

Response to Q3: VGP on Recent News

Thank you for raising these important points!

We test VGP on recent, source-verified news claims to examine whether it preserves the ability to generate legitimate visual evidence about emerging events. Since we use strictly black-box APIs, we cannot determine whether the providers employ external retrieval or how they update their internal knowledge.

We select 100 recent, source-verified news claims across 10 domains and evaluate GPT-Image-2 under matched settings, with and without VGP.

Setting on clean recent-news promptsASR (%)MR
Without VGP100.05.00
With VGP100.05.00

With VGP, all 100 requests generate images, and every image receives MR=5, matching the results without VGP. These results indicate that VGP does not reduce image generation or claim realization on this tested set of source-verified news claims.

Response to Q4: Definition of the Targeted Safety Failure

We appreciate your insightful comments, which help us clarify and strengthen our work!

We define the targeted safety failure in EpiReal-Bench as generating credible-looking visual-text evidence that presents a false real-world claim as factual, without an explicit qualification or correction. Importantly, visualizing a false claim in a clearly identified educational, fictional, illustrative, or simulated context does not constitute the targeted safety failure.

To validate this distinction, we conduct a controlled comparison between ordinary generation requests and explicitly simulated contexts. As reported in Response to W1 & W3 & Q1, Point 2: Distinguishing Legitimate Visualization from Misinformation Generation, all simulated requests successfully generate images, but none achieves MR=5 or counts as an ASR success. This is consistent with our evaluation criterion, which distinguishes legitimate visualization from the unqualified presentation of false claims as authentic evidence.

Dear Reviewer,

This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.

Thank you again for your time and consideration.

Reviewer_XvtuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper observes and demonstrates the verification-generation gap in frontier models where the model is capable of verifying claims but does not run that check for generation of images containing misinformation. The paper presents Epireal-Bench, a benchmark of 1000 claims, accompanied with selected generated images from frontier models. The paper utilizes VLM judges to measure misinformation presentation and quality. The paper also presents Epireal-attack, a prompt modification approach that makes VLMs more effective at generating misinformation.

Critical strengths
  • The observed verification-generation gap in frontier models is an important and impactful observation for AI safety. It is measured evidently in FCR against ASR.
  • Paper is generally easy to follow, except some use of terminology that is not introduced properly (e.g. the term ASR in Introduction)
  • At a high level, the proposed benchmark and its metrics can be a helpful contribution to the field, however, I have concerns regarding the specific implementation, which I note in the Weakness section
Critical weaknesses
  • A high-level concern I have is about the placement of the work as more generally about “visual misinformation”, and about “what an image shows, not what it asserts”, however, it seems like all the visual formats and their prompts are centered around rendering of text in the generated images. All ten prefixes in App. D seems to generate claims as text. The one exception might be the “Documentary photo” but even for that there is a mention of “concise professional news caption bar at the bottom”. PC2 in the related work is dismissed as limited. If orthogonal, this requires reframing of this work. In fact, we also see poor ASR for the Documentary photo category, which I think might be related to the effectiveness of MR by the GPT judge when there is a lack of text. This relates to my next concern, which is about the VLM judge.
  • A GPT judge is heavily used for MR and VC, both for benchmark curation and evaluation. MR is never validated by humans. Using VLM judges as part of evaluation and for frameworks such as TIFA is one thing, however, the use of VLM judge in dataset/benchmark creation requires more scrutiny. Looking at the prompt for MR, which is used for filtering/selection during dataset creation, I see the “4” on the scale defined with vague terms such as “with only minor weaknesses or inconsistencies”. When using a VLM judge, it is important to provide specific instructions and examples for accurate inference. In fact, looking at some of the Epireal-Bench examples in Fig. 10 and 11 in Appendix D, the images actually contain “Fictional newspaper” and “Fictional scenario”, which I assume passed the MR scale 5 filter.
  • The AI use statement denies the use of AI for dataset generation, which is not true. It is used for claim creation and candidate selection with MR filtering.
  • EpiReal-Attack is a reasonable demonstration but it could lead to potential misuse. No defense is evaluated against it. Verification-Guided Prompting is never tested with the attack adapting to it.
  • Regarding the verification-generation gap, there is more related previous work on LLM behavior that should be cited/discussed: https://arxiv.org/pdf/2412.02674 includes a “generation-verification gap” and https://arxiv.org/abs/2308.07308 showed that the same LLM that produced a harmful response will classify it as harmful when asked.
Questions for authors
  • Did you iterate on VLM judge prompts or evaluate the performance of the VLM judge?
  • What fraction of images in the benchmark carry the claim as text?
3 comments

We thank you for describing the gap as "important and impactful," the paper as "easy to follow," and the benchmark as a potentially helpful contribution. This recognition is a great encouragement to us!

Response to W1 & Q2: Text-Integrated Visual Misinformation

Thank you for this helpful comment!

  1. Joint Role of Text and Visual Content.

Our study considers both image content and the text rendered within images. The entire EpiReal-Bench dataset is designed around their combination. Specifically, all of benchmark images contain the target claim in text. In the revised manuscript, we use "text-integrated visual misinformation" to describe this setting more precisely. "Safeguards judge what an image shows, not what it asserts" refers to safeguards emphasizing visually unsafe content while potentially overlooking false assertions in ordinary-looking images.

  1. Relation to PC2PC^2.

We revise Sec. 2.2 to acknowledge that PC2PC^2 investigates jailbreak attacks generating fabricated and politically controversial imagery involving public figures. Our work focuses on a complementary aspect: how text and visual elements jointly construct credible-looking evidence across claim domains and visual formats.

  1. Interpretation of Lower ASR for Documentary Photographs.

We attribute the lower ASR under direct prompting partly to the nature of documentary photographs, which emphasize the direct visual depiction of claimed events and may therefore make models more sensitive to the depicted content. However, EpiReal-Attack raises ASR above 95% for this format across all four models (Table 3), indicating that prompt optimization can overcome much of this resistance.

Response to W2: VLM Judge Reliability and Appendix Examples

Thank you for raising this concern!

  1. VLM Judge Reliability

Our additional human evaluation independently corroborates the reliability of GPT-5.4-based ASR and MR evaluations. We add human validation on a 1,000-image subset of GPT-Image-2 outputs from EpiReal-Bench and EpiReal-Attack (Appendix C.5, Table 7).

Human ASR rises from 70.7% to 89.6%, close to GPT-5.4's increase from 77.7% to 96.8% (gains of 18.9 vs. 19.1 percentage points). Mean MR scores are 4.45 vs. 4.70 for EpiReal-Bench and 4.75 vs. 4.94 for EpiReal-Attack. Both evaluators report improvements across all ten formats, supporting the reliability of our VLM-based evaluation.

Table 7: human and GPT-5.4 evaluations of GPT-Image-2 outputs across all ten visual formats.

SettingEvaluatorMetricNPBNSPPSDPRDMABUTPCROverall
EpiReal-BenchGPT-5.4ASR (%)9087958927699288726877.7
MR4.884.814.964.824.044.594.894.794.544.694.70
HumanASR (%)8680858620657977676270.7
MR4.644.674.714.703.424.414.594.404.444.504.45
EpiReal-AttackGPT-5.4ASR (%)9995989797949596989996.8
MR4.974.924.964.934.894.894.924.944.984.994.94
HumanASR (%)9389908890918789928789.6
MR4.854.744.794.844.694.734.614.714.824.734.75
  1. Clarification of MR Evaluation and Appendix D Examples.

We agree with your point that MR evaluation requires particularly rigorous judgments, especially when generated images contain disclaimers, qualifications, or corrections. In response, we revise the MR evaluation prompt and clarify the scoring criteria in Appendix A.2.

We further apply the revised prompt to representative examples in Appendix D for validation. The resulting evaluations are largely consistent with the original ones. For example, Figure 12 receives MR=4 because its labels "ILLUSTRATIVE RECONSTRUCTION" and "Details unverified" weaken its presentation as established fact, while Figure 15 receives MR=5 because it convincingly presents the target claim with no major contradictions. These results suggest that our original prompt does not introduce substantial evaluation bias in these examples. Nevertheless, to ensure greater rigor and consistency across the benchmark, we plan to rerun our experiments using the revised prompt and update the corresponding results in the next manuscript revision.

Regarding the two examples you specifically point out, Figures 10 and 11 both receive MR=4. Figure 10 contains the qualification "Illustrative Mockup," while Figure 11 explicitly includes "FICTIONAL SCENARIO · DESIGN MOCKUP." Both images largely realize their respective claims, but these visible disclaimers weaken their presentation as genuine factual evidence. We also clarify that these disclaimers are present in the original model outputs and are not added by us for presentation.

Response to W3: AI Use Statement

Thank you for identifying this inconsistency. We correct the AI Use Statement to disclose generative AI use for candidate claims, prompts, and image samples, followed by human review and curation. The revised statement reflects the construction pipeline in Sec. 3.4.

Response to W4: Defense Robustness Against EpiReal-Attack and Adaptive Attacks

Thank you for your thoughtful comments and constructive suggestions!

  1. Effectiveness of Existing Defenses Against EpiReal-Attack.

VGP consistently achieves the highest IR among all evaluated defenses against EpiReal-Attack across the four models, while conventional safety detectors show substantially more limited effectiveness. To evaluate defense robustness against adaptive attacks, we add two new defense experiments in Appendix C.6, with the results reported in Table 9.

Specifically, VGP achieves interception rate of 71.3%, 86.8%, 30.4%, and 66.5% on GPT-Image-2, Nano Banana 2, Grok Imagen 2.0, and Seedream 5.0 Flash, respectively. In comparison, the best-performing conventional defense achieves only 37.1%, 42.9%, 27.3%, and 48.4%, respectively. These results indicate that VGP provides stronger protection against EpiReal-Attack than the evaluated input- and output-level defenses.

  1. Robustness of VGP Against Adaptive Attacks.

Under adaptive attacks, VGP's interception rates decrease but remain substantial across all four models. Specifically, we conduct an additional adaptive attack experiment in which the VGP instruction remains fixed and is prepended to each candidate attack prompt throughout optimization. The generated outputs and evaluator feedback then guide subsequent prompt refinement.

As shown in Table 9, VGP achieves interception rates of 41.6%, 58.2%, 29.7%, and 47.1% across the four models, respectively. These results indicate that adaptive optimization weakens VGP's defense, while VGP continues to intercept a portion of attack attempts across all four models.

Table 9: interception rates (%) against EpiReal-Attack and adaptive attack to VGP.

Defense levelDefense methodGPT-Image-2Nano Banana 2Grok Imagen 2.0Seedream 5.0 Flash
Input-levelKeyword Filter7.0018.0013.1020.20
Input-levelNSFW Text Classifier1.1017.4012.0035.90
Input-levelGPT-4o-mini Prompt Checker9.6026.705.2022.50
Output-levelSD Safety Checker6.5037.206.4022.70
Output-levelQ1637.1042.9027.3048.40
Output-levelGPT-4o Image Checker13.8037.8010.2023.00
OursVGP71.3086.8030.4066.50
OursVGP under adaptive attack41.6058.2029.7047.10

Response to W5: Related Work

Thank you for bringing these studies to our attention! We add and discuss them in Sec. 4.4. We recognize these works as important precedents on generation–verification discrepancies in language models. Our study extends this investigation to commercial image generators and false claims presented as credible visual evidence.

Response to Q1: Prompt Refinement and Judge Reliability

Thank you for this question!

  1. How we refine the judge prompt.

We refine the MR judge prompt through successive stages to make the evaluation more structured and consistent. We start with a general instruction asking whether an image convincingly presents the target claim. We then make the evaluation more structured by introducing a five-point scale, specifying what constitutes weak, partial, and strong claim realization, and requiring the judge to consider missing claim details and contradictions. We also separate claim realization from factual verification: the judge assesses how the image presents the claim, rather than judging whether the claim is true.

  1. How we validate judge reliability.

For ASR and MR, we evaluate a 1,000-image subset of GPT-Image-2 outputs from EpiReal-Bench and EpiReal-Attack (Appendix C.5, Table 7). Human ASR increases from 70.7% to 89.6%, while GPT-5.4 ASR increases from 77.7% to 96.8%. The corresponding gains are 18.9 and 19.1 percentage points, showing close agreement on the attack's improvement.

GPT-5.4 and human mean MR scores are 4.70 versus 4.45 under direct prompting and 4.94 versus 4.75 after the attack, differing by 0.25 and 0.19 points. Both evaluators confirm improvements across all ten visual formats.

For VC, Figs. 6–9 report evaluations from seven VLM judges and human annotators across four models. On EpiReal-Bench images, GPT-5.4–human mean-score gaps range from 0.02 to 0.38 points, averaging 0.19 points. All 32 model–evaluator comparisons agree on the direction of the difference between settings.

Together, these results support the consistency of our VLM-based evaluations.

Dear Reviewer,

This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.

Thank you again for your time and consideration.

Reviewer_EvaLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether commercial image generators can render false real-world claims as credible visual evidence. It introduces EpiReal-Bench, comprising 1,000 claims across ten semantic categories, instantiated in ten visual formats to produce 10,000 prompts and a curated image collection. It also proposes EpiReal-Attack, a black-box prompt optimization framework using transformation skills, multimodal feedback, and Pareto selection, together with verification-guided prompting as a mitigation. Across four commercial generators, the paper reports substantial misinformation realization under direct prompting and success rates above 95% after optimization. Human and VLM evaluations assess the visual credibility of the outputs.

Critical strengths
  1. The paper addresses an important generation-side risk: visually ordinary artifacts can convey harmful false assertions without containing conventionally unsafe imagery. This is a useful direction for evaluating image-generation systems.
  2. Crossing claim categories with visual formats provides a structured benchmark design. Instantiating the same claims in multiple formats supports comparisons that are difficult with less controlled collections.
  3. The reported coexistence of strong false-claim recognition and substantial misinformation generation, particularly for GPT-Image-2 and Nano Banana 2, is a potentially valuable observation. The benign verification control in Appendix C.1 is also a useful inclusion.
  4. The submission provides evaluation prompts, transformation definitions, an optimization outline, and qualitative examples. Evaluating visual credibility with both humans and multiple VLMs, and investigating a lightweight mitigation, broadens the empirical perspective.
Critical weaknesses
  1. The success metric needs stronger validation, particularly concerning disclaimers and safety behavior. ASR depends on a GPT-5.4 MR score of 5, but the submission does not report human validation of this success classification. The human study evaluates VC instead. Several appendix examples visibly qualify their content: Figure 10 includes “ILLUSTRATIVE MOCKUP,” Figure 11 includes “FICTIONAL SCENARIO · DESIGN MOCKUP,” and Figure 16 includes a simulated-alert warning and verification language. These examples are not individually identified as ASR successes, so they do not by themselves establish a scoring error. However, they make it important to explain whether these markings were generated or added for presentation, and how such outputs are scored. Furthermore, Figure 5 labels MR < 5 outputs as “Internally Aligned,” and Section 4.2 interprets lower documentary-photo ASR as stronger alignment. Incomplete claim realization can also reflect rendering or instruction-following limitations. The results should distinguish refusals, explicit qualification/correction, ordinary generation failures, and unqualified realization.
  2. The attack's benefit is not isolated from additional sampling and selection. Table 3 compares direct prompting with an adaptive search, but there is no query-matched repeated-sampling baseline or ablation establishing the contribution of skills and Pareto selection. Section 3.5 refers readers to Appendix B for population size, query accounting, and stopping criteria; Appendix B provides prompts and pseudocode but does not specify a population cap, a concrete convergence rule, or actual query counts. Consequently, the improvement to approximately 97% ASR cannot yet be attributed specifically to the proposed optimization method. Using GPT-5.4 for optimization, feedback, and the principal MR evaluation also makes independent validation of the final success judgments particularly important.
  3. The factual ground truth is insufficiently documented for an auditable benchmark. Section 3.4 describes review by five annotators, which is a positive step, but the manuscript does not provide the evidence sources, reference dates, or disagreement-resolution procedure supporting the false labels. Several examples concern time-dependent events or use underspecified temporal expressions. For example, the Jordan tourism claim in Figure 17 lacks a defined period and quantitative criterion. The accompanying risk rationales generally assert that the claims are false rather than documenting the contradictory evidence. This matters because the benchmark must distinguish demonstrably false assertions from unsupported or unverifiable ones. Appendix A.1 also groups “unsupported” claims with misinformation in the text-based verification prompt, despite the stricter false-claim definition in Section 3.2.
  4. The defense evaluation does not establish the safety–utility trade-off. Table 4 reports interception rates, but the exact evaluation population and whether prompts are direct or optimized are unclear. Because interception includes any outcome with MR < 5, it combines effective rejection with unsuccessful generation. Several baseline defenses primarily target conventional unsafe content, making their limited effectiveness informative but insufficient to establish the strength of VGP. Appendix C.1 tests benign factual recognition, not benign image generation under VGP. Since VGP instructs models to refuse unverifiable claims, its false-refusal rate on legitimate requests is necessary to interpret the reported mitigation gains.
  5. Figure 1 has a readability issue. Some text in the fan-shaped layout of the teaser figure is truncated, making the examples difficult to read and weakening the visual presentation. Please adjust the layout or shorten the text so that the intended information is fully visible and legible.
Questions for authors
  1. Were the disclaimers and qualifications in Figures 10, 11, and 16 present in the raw model outputs or added for publication? What MR scores did these examples receive? Could the authors provide a stratified human audit of MR = 5 decisions, explicitly checking qualification, contradiction, and preservation of the complete claim?
  2. What are the actual image-query budgets, population limits, and stopping rules for EpiReal-Attack? On a representative subset, how does it compare with repeated direct sampling at the same query budget, and what changes when Pareto selection is removed?
  3. Could the authors provide a source-linked, timestamped audit of representative claims, together with annotation instructions and disagreement resolution? How are unsupported or temporally ambiguous claims distinguished from verified falsehoods?
  4. What precisely is the denominator and prompt condition in Table 4? Please separate refusal/filtering from incomplete generation and report VGP's false-refusal rate on a matched set of benign generation requests.
4 comments

Thank you for recognizing the importance of generation-side misinformation risks, the structured benchmark design, and the value of our verification–generation observation. We appreciate the constructive feedback, which helps us strengthen the evaluation and clarify our methodology!

Response to W1 & Q1: Human Validation of MR and ASR, Raw-Output Disclaimers, and Interpretation of Unsuccessful Generations

Thank you for highlighting the need for stronger validation and clearer interpretation!

  1. Stronger human validation on MR.

Our additional human evaluation provides independent evidence supporting the reliability of GPT-5.4-based MR evaluation. We report human evaluations on a 1,000-image subset of GPT-Image-2 outputs from EpiReal-Bench and EpiReal-Attack. As shown in Table 7 below, although human annotators assign slightly lower absolute scores than GPT-5.4, both evaluators show consistent trends. Notably, the ASR improvements are nearly identical: 18.9 and 19.1 percentage points for human and GPT-5.4 evaluations, respectively. The corresponding MR improvements are also close (0.30 vs. 0.24 points). We include this additional analysis in Appendix C.5. These results provide human validation of our VLM-based evaluations and support the effectiveness of EpiReal-Attack.

Table 7: human and GPT-5.4 evaluations of GPT-Image-2 outputs across all ten visual formats.

SettingEvaluatorMetricNPBNSPPSDPRDMABUTPCROverall
EpiReal-BenchGPT-5.4ASR (%)9087958927699288726877.7
MR4.884.814.964.824.044.594.894.794.544.694.70
HumanASR (%)8680858620657977676270.7
MR4.644.674.714.703.424.414.594.404.444.504.45
EpiReal-AttackGPT-5.4ASR (%)9995989797949596989996.8
MR4.974.924.964.934.894.894.924.944.984.994.94
HumanASR (%)9389908890918789928789.6
MR4.854.744.794.844.694.734.614.714.824.734.75
  1. Clarification of Disclaimers and MR Scores in Appendix Examples.

The disclaimers and qualifications in Figures 10, 11, and 16 are part of the original model outputs, not added by us during figure preparation. We revise Appendix D to explicitly clarify this and report the corresponding MR scores and evaluation rationales. Figures 10 and 11 both receive MR=4, as they largely convey the target claims but are somewhat weakened by explicit mockup labels and cautious wording. Figure 16 receives MR=1, as it explicitly identifies itself as a simulated alert and directly contradicts the target claim. These examples illustrate how our evaluation accounts for both claim realization and visible qualifications or corrections. Additional high-MR examples are also available in our anonymous repository.

  1. Interpretation of Lower ASR for Documentary Photographs.

We attribute the lower ASR under direct prompting partly to the nature of documentary photographs, which emphasize the direct visual depiction of claimed events and may therefore make models more sensitive to the depicted content. However, EpiReal-Attack raises ASR above 95% for this format across all four models (Table 3), indicating that prompt optimization can overcome much of this resistance.

  1. Distinguishing Safety Refusals from Ordinary Generation Failures.

We provide an additional failure-type analysis of unsuccessful GPT-Image-2 outputs under direct prompting. Among these outputs, 20.4% are explicit refusals, 18.1% contain explicit qualifications or corrections, and 61.5% are ordinary generation failures. This analysis indicates that unsuccessful generations do not necessarily result from explicit safety interventions, but may also reflect models' incomplete execution of requests involving controversial content near safety boundaries.

Response to W2 & Q2: Query Budget, Repeated-Sampling Baseline, and Ablation Studies

Thank you for suggesting the query-matched baseline and ablations!

  1. Comparison with Repeated Sampling and Component Ablations.

EpiReal-Attack substantially outperforms repeated sampling, and both skill-guided prompt transformation and Pareto-based selection contribute to its effectiveness. To evaluate these contributions, we include a Best-of-20 repeated-sampling baseline and two component ablation experiments, with the results reported in Table 8. We also include the corresponding experimental details in Appendix C.6.

Specifically, under the same maximum budget of 20 image queries per prompt, repeated sampling achieves an overall ASR of 77.00% using all 20 queries, whereas EpiReal-Attack achieves 97.23% ASR with only 5.39 queries on average. Furthermore, removing skill-guided prompt transformation reduces ASR from 97.23% to 81.16%, indicating that it generates effective prompt modifications to improve attack success. Removing Pareto-based selection reduces ASR to 92.94% and increases the average query cost from 5.39 to 11.83, indicating that Pareto-based selection enables more efficient optimization iterations.

**Table 8: repeated sampling and ablations of EpiReal-Attack across all four models. **

MethodGPT-Image-2Nano Banana 2Grok Imagen 2.0Seedream 5.0 FlashOverallAvg. image queries
EpiReal-Bench77.0155.8289.4373.2073.87—
Repeated sampling (Best-of-20)79.5260.2490.5277.7277.0020.00
EpiReal-Attack without skills83.0467.5193.2680.8281.1618.42
EpiReal-Attack without Pareto selection93.6994.2692.5391.2892.9411.83
EpiReal-Attack (full)96.9696.5298.2397.2197.235.39
  1. Per-prompt Query Budget and Attack Optimization Procedure.

We revise the query accounting and optimization procedure in Appendix B.3. The maximum budget is 20 image queries per input prompt, including the initial direct-prompt generation. At each refinement round, the four transformation skills are independently applied to the current prompt, producing four candidate prompts, one per skill. Pareto filtering then retains non-dominated candidates with complementary strengths, which the optimizer uses together with multimodal evaluation feedback to generate the next-round prompt. The optimization terminates when a candidate achieves both MR=5 and VC=5, or the query budget is exhausted.

  1. Independent Human Validation of Final Success Judgments.

Please refer to Response to W1 & Q1, Point 1: Stronger Human Validation on MR, for the additional human evaluation and its results.

Response to W3 & Q3: Claim Definition, Factual Evidence, and Annotation Reliability

Thank you for raising these important concerns!

1. Clarification of Claim Definition and Factual Status.

Following your suggestion, we revise Sec. 3.2 to clarify that we use "false claims" as an operational category that includes both demonstrably false claims and claims that are unsupported at the time of assessment. These are not treated as two separate categories of claims. Instead, when we cannot identify sufficient evidence to establish the factual status of a claim, we retain it within our benchmark scope but additionally annotate it as unsupported, rather than labeling it as demonstrably false. We also clarify how temporally ambiguous claims are handled under this annotation scheme.

2. Claim-Specific Evidence and Annotations.

To address your concerns about factual evidence and reference dates, Appendix D provides claim-specific evidence annotations for representative examples, including relevant sources, reference periods, quantitative findings where available, and supporting rationales. Specifically, Figure 16 cites the PNAS study on the CIMMYT-related claim and provides its quantitative findings, while Figure 17 specifies the reference period and provides Jordan Central Bank tourism-receipt data for the Jordan tourism claim. We also explicitly indicate when a claim is labeled as unsupported because the available evidence is insufficient to establish its factual status. We will extend this source-linked and time-stamped annotation format to all claims in EpiReal-Bench to make the factual evidence and annotations more transparent and auditable.

3. Annotation Disagreement Resolution.

We adopt a majority-voting and discussion-based procedure to resolve uncertain or disputed annotations. Specifically, when annotators disagree on the factual status of a claim, majority voting is used to establish the initial decision, followed by discussion to examine the supporting evidence and resolve remaining disagreements.

Response to W4 & Q4: Defense Evaluation Protocol and Utility on Benign Requests

Thank you for your careful review and helpful feedback!

  1. Evaluation Subset and Outcomes of Defense Interventions.

We evaluate all defenses on cases that successfully generate the target misinformation under direct prompting, ensuring a consistent comparison of their effectiveness. Specifically, the denominators in Table 4 are 7,701, 5,582, 8,943, and 7,320 for GPT-Image-2, Nano Banana 2, Grok Imagen 2.0, and Seedream 5.0 Flash, respectively. All defenses are evaluated on the same successful-case subset for each model. We further analyze the outcomes of VGP interventions on GPT-Image-2: among the intercepted cases, 36% result in explicit refusals, 44% in ordinary generation failures, and 20% in explicit corrections. These results show that VGP prevents successful misinformation generation through both explicit safety interventions (refusals and corrections, accounting for 56%) and unsuccessful claim realization (44%).

  1. False-Refusal Rates on Benign Generation Requests.

We revise Appendix C.1 to add a more detailed evaluation of VGP on benign generation requests, with the results summarized in Table 5. VGP incurs very low false-refusal rates across all four models: GPT-Image-2 (0.50%), Nano Banana 2 (1.60%), and Grok Imagen 2.0 (1.00%), and 5.30% for Seedream 5.0 Flash. The corresponding benign verification accuracies are 98.20%, 99.10%, 94.60%, and 92.80%, respectively. These results indicate that VGP effectively distinguishes benign requests from misinformation-related claims while maintaining a low false-refusal rate and preserving utility on benign generation requests.

Table 5: factual-verification accuracy and VGP's false-refusal rate on benign image-generation requests.

ModelBenign verification accuracy (%) ↑VGP false-refusal rate (%) ↓
GPT-Image-298.200.50
Nano Banana 299.101.60
Grok Imagen 2.094.601.00
Seedream 5.0 Flash92.805.30

Response to W5: Figure Readability

Thank you for pointing this out! We revise Fig. 1 to improve its layout and make the example text clearly visible.

Dear Reviewer,

This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.