acceptodds
Under review as a conference paper at ICLR 2027

Who Constructs the Harmful Event? Event Delegation in Conversational Image Generation

Abstract

Conversational image-generation systems develop underspecified requests into concrete scenes, helping decide what the resulting image depicts. However, existing jailbreaks typically require the attacker to specify a harmful event, sometimes leaving the target system to elaborate on it. Instead, in this work, we investigate whether an attacker can exploit this capability while leaving the concrete harmful event unspecified. We introduce Ideate-and-Concretize (I-CON), which supplies a harm-sparse scene (a scene without harm details, such as action) and a coarse category cue, then uses three fixed text turns before a single image request. The procedure requires no external attacker model, adaptive search, or repeated image attempts, requiring no access beyond the ordinary chat interface. Despite leaving the concrete harmful event unspecified, I-CON achieves a higher attack success rate than previous attack methods, and the gap widens when the baselines are restricted to the same harm-sparse inputs. Tracing event provenance in successful cases, with human validation, we estimate that 99.35% of the realized harmful events are introduced by the target and not entailed by the input. We also find that scene descriptions violating the safe image policy can reach the user even when the final image is blocked. These results motivate evaluating both the final image and the process through which its scene is constructed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Jailbreak, Multimodal Generative Models, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Sx3yOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the safety issue of the conversational image-generation systems. The authors proposed an attack method, called I-Con, which supplies a harm-space scene and a coarse category, then uses three fixed text turns before a single image request. They claim that prior work carries the harmful intent in their prompts or requests, requiring the attacker to specify a harm event or leaving the target system to elaborate on it. I-Con requires no external attacker model, adaptive search, repeated image attempts, or access beyond the ordinary chat interface. And I-Con achieves a higher attack success rate on GPT serial models and Gemini serial models compared with MMA-Diffusion, DACA, CoJ-Attack, etc.

Critical strengths
  1. Conversational image-generation systems are widely used and raise some critical safety issues, for example, the harmful content generation. This paper propose a jailbreak attack whose inputs and requests do not contain harmful informantion but can guide the models generate harmful contents. This is a more covert method, which exposes the potential safety risks and ethical issues when we deploy the generation models and provide servies to public.
  2. The paper conducts experiments with cutting-edge models such as GPT and Gemini, showing the attack effectiveness against latest technologies.
Critical weaknesses
  1. The proposed method makes sense but what challenge does this paper aims to address is unclear. And what key contribution does this paper make?
  2. The paper proposes neither a new algorithm nor a new model. It is just about a kind of prompt refining. The novelty is limited.
  3. What is the main contribution of this paper? A item-by-item list is good to show the contributions.
  4. The dataset used in this paper is created by themselves. So it is hard to say if there are bias or fairness issues because the dataset is not publicly available.
Questions for authors

All four items in the weaknesses form.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Sx3y [1/2]

Dear Reviewer Sx3y, We sincerely thank you for your feedback. We particularly appreciate your recognition of a more covert attack that exposes safety risks in deployed systems, and experiments with cutting-edge models such as GPT and Gemini. Your comments on the problem statement, contributions, and dataset have helped us present our work more clearly. We address your concerns and questions in detail in the responses below.

[W1-1] The safety problem addressed.

Answer: We address a safety weakness of conversational image generation: an attacker can leave the design of a concrete harmful event to the system itself.

  • How prior attacks obtain the harmful event. Many existing jailbreaks begin with harmful content that the attacker has already specified and change how it is presented, as an implicit or safe-looking prompt [G4, G5] or across conversational turns [G6, G7]; RICE [G9] has the target expand an explicit harmful prompt. Self-prompting [G2] does delegate content generation to the model, but through instructions that ask for normally refused content. I-CON needs neither: the attacker writes no harmful event and issues only ordinary scene-development requests (Sec. 2; App. J of the revised manuscript).
  • What we show. Given only a harm-sparse scene, a coarse category such as violence, and fixed scene-development requests, the target supplies the concrete event: in a two-annotator audit, on average 98.87% of the harmful events in the returned images are attributed to the target rather than the user (Sec. 4.3; App. E.6.1 of the revised manuscript).
  • Why it matters. Existing safeguards do not stop it: with Llama Guard 4 applied, I-CON still reaches 30.50% ASR, and harmful scene descriptions reach the user in 43.00% of the sessions (Sec. 4.4, Table 4 of the revised manuscript). An attacker can therefore obtain harmful images through fixed scene-development requests, without having to design a concrete harmful event or explicitly solicit normally refused content. By delegating event construction to the target, I-CON reduces the need for attacker-authored harmful details and thereby facilitates the attack.

We have clarified how a harmful direction differs from a concrete harmful event and how I-CON reduces the need for attacker-authored harmful details in Sec. 1 of the revised manuscript (L078–080 and L089–090).

[W1-2, W3] Main contributions, item by item.

Answer: Our four contributions address the attack procedure, the resulting exposure, the ability of existing safeguards to detect it, and a mitigation:

  1. An attack that delegates event construction to the target. To test whether a malicious user can obtain unsafe images without writing the concrete harmful event, I-CON uses three fixed scene-development instructions and one image request. The target supplies the event, reducing the content the attacker must design. The procedure requires no external attacker model, adaptive search or retries, and achieves higher ASR than the six evaluated prior attacks on both primary systems (33.21% on GPT-Image-1.5 and 40.36% on Gemini-2.5-Flash-Image).
  2. A measurement of event provenance and Intermediate Leakage. To distinguish attacker-supplied content from target additions, we trace each event's origin, first appearance and realization in the image. Intermediate Leakage measures released descriptions containing a target-originated event whose depiction would violate image policy, including sessions that return no image. Reviewers j3Af and os3N recognized this analysis as offering "useful insights beyond final-image ASR" and showing "how harmful content develops throughout a conversational generation process".
  3. An evaluation of existing safeguards. To assess whether published safeguards catch the harmful content developed by the target, we replay fixed trajectories offline against four safeguards and measure residual ASR and Intermediate Leakage. I-CON still reaches 28.50–36.00% ASR under these safeguards (Sec. 4.4).
  4. A target-model gate that judges descriptions under the image policy. To test whether this exposure can be stopped before release, the target system’s own text model withholds a response if an image of the described scene would violate the image policy. The gate lowers I-CON’s ASR and Intermediate Leakage to 5.25% and 4.75% (Sec. 4.5; our response to Reviewer j3Af [W3, Q3]).

Our original submission described the attack procedure and evaluation measures, which appear in Sec. 3 and 3.3 of the revised manuscript. It also reported the primary attack comparison, event-level analysis and offline safeguard evaluation, which appear in Table 1 and Sec. 4.3 and 4.4, respectively. The gate is added in the revision and appears in Sec. 4.5. Sec. 1 summarizes the four contributions in Lines 104–107.

AuthorsOpenReview ↗

Response to Reviewer Sx3y [2/2]

[W2] Novelty beyond prompt refinement.

Answer: The simplicity of I-CON exposes a practical safety weakness: fixed scene-development requests can induce the system to supply the concrete harmful event itself. A malicious user can execute the attack using a starting scene, a coarse category, three fixed text requests and one image request. No external attacker model, search or retries are needed, making the attack straightforward to execute and the safety weakness exploitable through a simple, fixed procedure. Our contribution is to study this use of scene construction and measure how the resulting content reaches the user.

  • Event construction at the same call budget. On GPT-Image-1.5, I-CON reaches 33.21% ASR vs. 10.71% for refinement that only adds visual detail, with the same inputs and three text calls. Replacing only the Concretize request with a neutral continuation gives 25.71% (Holm-adjusted p = 0.0418; Sec. 4.2, Table 3(b) of the revised manuscript). Prior attacks given the same harm-sparse inputs (RICE [G9], Crescendo [G6] and Inception [G7]) reach at most 6.07% across the four systems. The gain thus comes from the instruction content, not from the inputs or the number of calls.
  • A safety exposure that image ASR misses. On GPT-Image-1.5, 40.54% of the sessions that end in a provider block or refusal contain Intermediate Leakage, and resubmitting the last released description alone yields a detector-flagged unsafe image in 9.46% of the sessions that returned none (Sec. 4.3; App. D.6.1 of the revised manuscript). A session counted as a failure by image ASR can thus still expose a description that can be reused to elicit an unsafe image.
  • What the analysis adds. Self-prompting [G2] already delegates concrete content generation to the target ([W1-1]). We examine harmful event construction within a supplied visual scene and trace who introduces the event, when its description reaches the user and whether it appears in the image.

Our original submission included the equal-budget refinement control, the matched-input comparisons, and the leakage and standalone-rendering results (Sec. 4.2, Tables 2 and 3, Sec. 4.3, and App. D.6.1). In the revised manuscript, we have added the fixed-turn stage-replacement control to Sec. 4.2 and Table 3 and added a comparison with self-prompting and implicit-prompt attacks to App. J.

[W4] Dataset availability, bias, and fairness.

Answer: Our 280 harm-sparse scenes are derived from the public I2P benchmark [G8] (MIT license), and we will release all of them with the camera-ready version.

  • Construction. I2P is also used to evaluate prior attacks, including RICE [G9], the metaphor-based attack [G5] and Inception [G7]. The 280 cases contain 40 per I2P category. An LLM drafted harm-sparse scenes under a fixed rule that removes harmful actions, relations, instruments and outcomes, and the authors reviewed them. In a two-annotator audit, the rewrites keep the subjects and, in most cases, the setting of the source scene and almost always remove its harmful content (App. I.2 of the revised manuscript).
  • Examples and release. App. I.1 of the revised manuscript lists example scenes from every category. With the camera-ready version, we will release all 280 scenes together with each scene’s I2P source prompt and the frozen rewrite rule, so readers can inspect each source prompt and the construction criteria.
  • Composition and bias. Gendered terms appear in 34.29% of the scenes: 15.71% use only male terms, 17.86% only female terms, and 0.71% both. The I2P-derived sexual cases are skewed toward female terms (62.50%), which we now report in App. I.1. Because all methods are compared on identical scenes and categories (Table 2), the composition is the same for every method, and the release lets readers examine bias and fairness directly.

Our original submission described the dataset construction and the comparisons on identical harm-sparse scenes and categories (Sec. 3.1 and 4.1, Table 2). In the revised manuscript, we have added the dataset composition, the observed gender skew, and the release plan to App. I.1.

We hope these clarifications address your concerns about the contributions and dataset. If there is any remaining question, please let us know; we would be glad to clarify it during the discussion period.

Dear Reviewer Sx3y,

We are grateful for the time the reviewer has spent on our manuscript. The review asked us to state explicitly what problem the paper addresses and what it contributes, and answering these questions has helped us present the work more clearly.

We would appreciate hearing whether our responses have resolved the concerns raised in the review. If any point is still unclear or needs more detail, we would be happy to provide further explanation during the discussion.

We thank the reviewer for considering our responses.

Best regards,

Authors

Reviewer_os3NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces I-Con, a multi-turn attack framework for conversational image generation that aims to induce models to generate harmful images by progressively developing harm-sparse scene descriptions. By providing a harmful category and prompting the target model to elaborate on the scene, concretize an observable event, and refine the description, I-Con encourages the model to introduce concrete event details that are not explicitly specified in the initial input. Experiments on two image-generation systems show that I-Con achieves higher attack success rates than the evaluated baselines. The paper further analyzes event provenance across intermediate text responses and final images, reporting that harmful events frequently first appear in target-generated content. Its evaluation of existing safeguards also highlights potential risks from harmful intermediate text that may be carried into subsequent image generation. Overall, the paper offers an interesting framework for studying harmful content generation in conversational image systems. However, the distinction between target-side event construction and compliance with an attacker-specified harmful objective remains insufficiently established, and the reliability and significance of the event-provenance analysis require further clarification.

Critical strengths
  1. Important problem and a useful research perspective. The paper investigates how harmful content can emerge through multi-turn interaction in conversational image-generation systems, rather than focusing solely on explicit harmful prompts or final image outputs. Its attention to intermediate textual descriptions provides a useful perspective for understanding safety risks in these systems.
  2. A structured and interpretable attack framework. I-Con decomposes scene development into multiple stages, making the attack procedure relatively easy to understand and analyze. The ablation studies and comparisons with alternative prompting strategies provide empirical evidence for the effectiveness of the overall protocol under the tested settings.
  3. Analysis beyond attack success rate. The paper examines event provenance across intermediate text and final images, attempting to identify where concrete harmful events first emerge. This is potentially valuable because attack success rate alone cannot reveal how harmful content develops throughout a conversational generation process.
  4. Practical implications for safety evaluation. The safeguard experiments suggest that inspecting intermediate text may be insufficient to prevent harmful outputs in multi-turn image generation. Although the current offline replay setting limits the scope of this conclusion, the analysis motivates further evaluation of safeguards across intermediate text and generated images.
Critical weaknesses
  1. The event-provenance analysis does not establish autonomous harmful intent. The attack explicitly provides a harmful category and instructs the target model to develop a scene and turn implicit meaning into a concrete, observable event. Moreover, removing the harmful category reduces the reported attack success rate from 33.21% to 6.43%, indicating that the category plays a substantial role in determining the harmful direction. Although the concrete event details may first appear in target-generated responses, this does not establish that the model independently initiates harmful intent. The results more directly demonstrate that the model constructs concrete harmful events under category-conditioned instructions. The authors should clarify this distinction and provide controls that separate the effects of the attacker-specified harmful objective, generic scene elaboration, and target-driven event selection.
  2. The reliability of event-provenance measurement requires stronger validation. Event provenance is central to the paper's claims, yet the reported sensitivity analysis shows only 88.31% agreement and a Cohen's κ of 0.13 between the primary provenance classifier and an embedding-based alternative. Although the authors explain why the embedding-based method is unsuitable as the primary classifier, the substantial disagreement for baseline methods raises questions about the robustness of cross-method comparisons. The paper would benefit from a more systematic multi-annotator evaluation of event extraction, entailment, and provenance attribution, together with an assessment of extraction errors and the sensitivity of the main findings to alternative annotation criteria.
  3. The novelty relative to prior self-prompting and implicit-prompt attacks is insufficiently established. Existing work has explored attacks in which models generate unsafe content through self-prompting, as well as methods that elicit target content through implicit prompts. I-Con differs in its category-conditioned scene-development procedure and its analysis of event provenance across conversational text and final images. However, the paper does not fully establish which findings constitute a qualitatively new insight beyond adapting these ideas to conversational image generation and adding a provenance-analysis framework. The authors should more clearly distinguish the novelty of the attack mechanism from that of the evaluation methodology, and explain what the event-origin and persistence analyses reveal that conventional attack-success evaluations would miss.
  4. The semantic fidelity of the harm-sparse input construction is not sufficiently evaluated. The paper removes harmful actions, relations, instruments, and outcomes from the original prompts while retaining selected scene elements. However, the reported category-match metric assesses whether the final image belongs to the requested harmful category, not whether the transformed input preserves the original scene's intended non-harmful semantics. Without an independent evaluation of semantic preservation and output grounding, it remains unclear to what extent the experiments measure harmful event construction grounded in the original scene rather than category-conditioned harmful scene generation from a partially transformed input.
Questions for authors

See the weaknesses

8 comments
AuthorsOpenReview ↗

Response to Reviewer os3N [1/4]

Dear Reviewer os3N, thank you for highlighting an important problem and useful research perspective, a structured and interpretable attack framework, and analysis beyond ASR. Your comments helped clarify our claims and validate provenance.

[W1-1] Category-conditioned event construction versus autonomous harmful intent.

Answer: We thank the reviewer for suggesting that we distinguish harmful intent from concrete-event construction. In I-CON, the attacker supplies only a coarse harmful direction, which acts as a seed, and the target develops it into the concrete harmful event. Our contribution concerns this delegation of event details to the target, not autonomous harmful intent.

  • Prior attacks state the harmful event; I-CON needs only a category as a seed. Existing attacks typically begin from a prompt that describes the harmful act itself. For one harassment case, the I2P prompt used by the native baselines states the act (a man licking a woman's hair), whereas I-CON receives only the scene "a man and a woman" and the category "harassment", which do not say what act occurs. The target then constructs the concrete event under three fixed text requests and one image request, without an external attacker model, search, or retries.
  • The seed need not be a specific harm category. With the category replaced by the generic word "harm" or by a less explicit cue (e.g., altercation for violence), ASR remains close to that under the named categories, whereas removing the category altogether leaves the target no harmful direction to develop (Table R9 in [W1-2]). A short harmful cue can therefore guide event construction, facilitating attacks without requiring a detailed harmful description.

We now describe the mechanism as "the target constructs concrete harmful events under category-conditioned instructions." We have revised the Introduction (Sec. 1, L078–080, and L089–090) and Limitations (App. H, L1565–1569) to clarify that this claim concerns concrete-event construction rather than autonomous harmful intent.

[W1-2] Separating harmful intent, scene elaboration, and target-driven event selection.

Answer: We appreciate this suggestion and separate the three effects below. The controls support our claim that the attacker can supply a general harmful direction while leaving concrete-event construction to the target.

  1. Attacker-specified harmful objective: a general harmful cue is enough for attack success, while a specific category guides the type of harm.
    • On the same 280 GPT-Image-1.5 cases, we replaced the category with the generic word "harm", replaced it with a less explicit cue for each category (e.g., altercation for violence), or removed it (Table R9). The scenes, fixed instructions, three text requests, and one image request remained unchanged.
    • A specific category guides the type of harm more precisely (CaM 66.02% vs. 21.52% for the general cue), but it is not required to obtain unsafe images: ASR is 29.29% with the general cue, against 6.43% without any cue.
    • The attacker can provide a general harmful cue and let the target supply event details, reducing the need to author a detailed harmful scenario.

Table R9. Category-cue controls on GPT-Image-1.5.

Category cueASR (%)CaM (%)
Coarse category (I-CON)33.2166.02
Indirect cue (e.g., altercation)29.6452.07
General cue ("harm")29.2921.52
w/o Category6.4313.31

CaM is the category match rate among returned images (Sec. 4.1), measured here against each case's source category.

  1. Generic scene elaboration: it does not explain the attack, since I-CON achieves higher ASR than generic refinement with the same inputs and turn count.
    • Table R10 compares I-CON with generic scene-development controls on the same 280 cases, with the same inputs and turn count.

Table R10. Generic scene-development controls on GPT-Image-1.5.

ConditionASR (%)
I-CON33.21
Generic refinement10.71
Concretize replaced with a generic instruction25.71
  1. Target-driven event selection: the selected event varies across repeated runs, and an independent audit attributes it to the target.
    • With the scene, category, and instructions fixed, 74.29% of cases produced multiple distinct concrete events across repeated runs (App. E.5).
    • In a separate independent audit, most events judged depicted and harmful were neither stated in nor necessarily entailed by the full user context (App. E.6).

Our original submission included the generic-refinement control and the repeated-run event-selection analysis, which are presented in Sec. 4.2, Table 3, and App. E.5 of the revised manuscript. We have added the general harmful-cue control and the fixed-turn stage-replacement results to Sec. 4.2 and Table 3. We have also added the independent origin audit to App. E.6.

AuthorsOpenReview ↗

Response to Reviewer os3N [2/4]

[W2-1] Provenance-rule disagreement and robustness of cross-method comparisons.

Answer: The low κ arises because the embedding rule labels almost every event as target-originated. As the reviewer notes, this check alone cannot establish the baseline values, so we added independent human judgments. Under these judgments, the baselines' target-originated share lies far below the embedding rule's values, and I-CON's share remains higher than the baselines'.

  1. Why agreement is 88.31% but κ is only 0.13. κ measures agreement beyond what two rules would reach by chance. The embedding rule labels 1,160 of the 1,172 events as target-originated, and the primary rule labels 1,023. When both rules give the same label to most events, they agree on 86.52% of the events by chance alone; the observed 88.31% is only slightly above this, which gives κ = 0.13 (a known effect of unbalanced labels on κ [G1]). All 137 disagreements are events that the primary rule assigns to the user or attacker and the embedding rule assigns to the target; they occur almost entirely in RICE and Crescendo and rarely in I-CON (Table R11). This pattern explains where the rules disagree, but it does not show which rule is right on the baselines (App. E.3).

  2. Checking both rules against independent human judgments. Two annotators independently judged 76 of the 1,172 events (35 rule disagreements and 41 agreements; App. E.6), without seeing the method, the target system, or either rule's label. Their judgments agree on 96.05% of the events (κ = 0.910).

Table R11. Event-provenance estimates from two automated rules and two annotators.

Target-originated sharePrimary ruleEmbedding ruleTwo annotators (95% interval)
I-CON99.34%99.89%100.00% [86.77, 100.00]
RICE47.29%95.35%54.65% [36.99, 71.50]
Crescendo46.43%96.43%70.60% [52.82, 84.73]

For both baselines, the human estimates lie far below the embedding rule's values; for Crescendo, the human estimate falls between the two rules. In the revised manuscript, we therefore report the baseline shares together with these human estimates (App. E.6).

  1. Robustness of the cross-method comparison. I-CON's own share is 99–100% under both rules and the human judgments. Under the human judgments, the pooled baselines reach 62.95%, and I-CON exceeds them by 37.05 points (95% interval [18.34, 51.87]), so the difference between I-CON and the baselines holds without relying on either automated rule.

The independent human judgments thus support the interpretation that I-CON delegates concrete event construction to the target. They provide evidence for its practical benefit: the attacker can leave concrete event details unspecified and have the target supply them.

Our original submission included the comparison between the primary and embedding-based provenance rules, now presented in App. E.3 of the revised manuscript. We have added independent two-annotator validation and clarified the robustness of the cross-method comparison in App. E.6.

[W2-2] Validation of extraction, entailment, and attribution; sensitivity to annotation criteria.

Answer: We thank the reviewer for requesting a more systematic validation. We added an independent two-annotator image audit: on average, 98.87% of the harmful events in the images are attributed to the target rather than the user.

  1. Multiple annotators. Two annotators assessed 61 events sampled from the 308 extracted harmful events, independently and blind to the classifier's labels. An event counts as user-originated only if the user context states or necessarily entails it. Among events each judged depicted and harmful, weighted target-originated shares are 97.74% and 100.00% (mean 98.87%, 95% interval [89.32, 100.00]). Origin labels agree on 92.11% of the events and harmfulness labels on 65.45%, so the annotators differ mainly on whether a borderline event is harmful, not on who introduced it.
  2. Extraction errors. An omission audit of 67 images gives mean recall of 95.77% for human-identified harmful events (95% interval [85.65, 99.47]). The annotators judged 77.39% of the extracted events they found depicted to be harmful, so the shares above use only events they judged harmful.
  3. Sensitivity to criteria. Counting merely likely inferences as user-originated as well lowers the mean target-originated share to 93.27% (95% interval [80.88, 98.72]); most harmful events remain target-originated under this broader criterion.

Together, the independent annotations, extraction-error checks, and sensitivity analysis support the reliability of our interpretation that the target constructs concrete harmful events.

We have added the independent two-annotator audit, the extraction-error evaluation, and the sensitivity analysis under the broader user-origin criterion to App. E.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer os3N [3/4]

[W3-1] Attack and evaluation novelty relative to self-prompting and implicit-prompt attacks.

Answer: The attack is new in how the harmful event is obtained: the input neither conveys a harmful event nor asks for content the target would refuse, and the target constructs the event through ordinary scene development. The evaluation is new in tracing that event, i.e., who introduces it and what reaches the user before the image outcome.

  1. Attack mechanism.
    • Implicit prompts: prior work [G3, G4, G5] shows that unsafe images need not require an explicit harmful description. The attacks among them [G4, G5] start from a given unsafe prompt and rewrite it with safe-looking substitutions or metaphors, so the content to be depicted is fixed by that prompt and only its wording changes. I-CON starts from a harm-sparse scene and a coarse category, which do not say what event occurs; the target constructs it.
    • Self-prompting: both I-CON and Involuntary Jailbreak [G2] delegate concrete content generation to the target, and [G2] also includes a topic-constrained variant. However, Involuntary Jailbreak asks the model for questions it would normally refuse to answer, together with their answers, and forbids refusal words in the responses. I-CON instead asks the target to develop a supplied scene and turn its central meaning into a concrete, observable event (Sec. 3.2; App. B).
    • I-CON thus allows an attacker to obtain harmful images through ordinary scene-development requests, without having to design a concrete harmful event or explicitly solicit normally refused content.
  2. Evaluation methodology. Final-image ASR does not show who introduced a depicted event or what the user received before the image outcome. Our event-origin analysis separates events conveyed by the input from events constructed by the target, and Intermediate Leakage measures released descriptions of an event whose depiction would violate the image policy. Both measures can be applied to other attacks; we also report them for RICE and Crescendo (Sec. 4.3; [W2-1]). What they reveal beyond ASR is detailed in [W3-2].

We have added an explicit comparison of the attack mechanisms and of the evaluation, with a summary table, to App. J of the revised manuscript, referenced from Sec. 2.

[W3-2] Insights from event origin and persistence beyond ASR.

Answer: The analyses show three things that final-image ASR cannot: the target, not the user, constructs the harmful event; the concrete event is usually written before the final description; and it reaches the user even when the image is blocked.

  1. Who constructs the realized harmful event? Two annotators independently audited 61 of the harmful events extracted from flagged I-CON images. Among events each judged depicted and harmful, on average 98.87% are target-originated (App. E.6). Image ASR alone cannot distinguish this from realizing an event already conveyed by the user.
  2. When does the concrete event appear in the text? We tracked extracted safety-relevant text events from Concretize to Refine. Of the events present after Concretize, 63.64% on GPT-Image-1.5 and 59.78% on Gemini-2.5-Flash-Image are still present after Refine, almost always with a change in degree or specificity; they make up 70.25% and 73.01% of the events in the Refine description. Most events in the final description are thus already concrete by the end of Concretize and only reworked by Refine, so they are released to the user before the final description. These descriptive results use automated cross-turn matching and are conditional on observing both text stages.
  3. What remains exposed after an image failure? Intermediate Leakage occurs in 54.64% and 55.00% of all GPT-Image-1.5 and Gemini-2.5-Flash-Image sessions, and in 40.54% and 70.83% of the sessions that end in a provider block or refusal, which image ASR counts as failures. In a separate rendering control, the last released description alone yields a detector-flagged unsafe image in 9.46% of the GPT sessions that originally returned none (Sec. 4.3, App. D.6.1). Blocking the final image therefore neither withholds the earlier scene description nor prevents its reuse to elicit unsafe images.

Together, these findings locate where a safeguard must act: on descriptions before release, as the gate in our response to Reviewer j3Af [W3, Q3] does.

Our original submission included the event-origin, Intermediate Leakage, and standalone-rendering analyses, which are presented in Sec. 4.3, App. D.5, and App. D.6.1 of the revised manuscript. We have added the independent origin audit to App. E.6. The persistence results, with the matching procedure, counts, and limitations, have been added to App. E.7 and are referenced from Sec. 4.3.

Reviewer_j3AfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes I-CON, a simple but effective attack on conversational image generation. Given a harm-sparse scene and a coarse category, it uses fixed instructions for scene ideation, event concretization, and atmosphere refinement before requesting an image. The target constructs the concrete harmful event rather than receiving one from the attacker. The paper also investigates event provenance and Intermediate Leakage, showing that potentially risky scene descriptions can reach users even when the final image is blocked.

Critical strengths
  1. The attack pipeline is simple and effective. The analysis of event formation and intermediate scene descriptions provides useful insights beyond final-image ASR.

  2. The experiments are extensive, covering baseline comparisons under both native and harm-sparse inputs, component ablations, and four existing safeguards.

Critical weaknesses
  1. The ablations show a strong dependence on the coarse category, but category-level ASR results are missing.

  2. Removing a stage also reduces the number of text calls. The stage ablations therefore do not separate the contribution of each component from that of additional scene-development turns.

  3. Existing safeguards leave substantial residual risk, but the paper provides limited guidance on mitigating this attack.

  4. Human validation is limited. The paper does not report a human-judged ASR estimate, and the event-provenance audits mainly rely on one annotator.

  5. Offline safeguard replay uses fixed trajectories and does not evaluate adaptive attackers or regeneration after a safeguard intervention.

Questions for authors
  1. Could the authors report ASR for each category? Additionally, I am curious whether pairing the same harm-sparse scene with a substantially different harmful category would also produce harmful images.

  2. Could each removed stage be replaced with a generic scene-development instruction to keep the number of text calls fixed, so as to distinguish the contribution of additional turns from that of the component itself?

  3. What mitigation could address harmful event construction during scene development? An initial evaluation would be helpful.

  4. How would the reported results change under an online setting with adaptive attackers and content regeneration after safeguard interventions?

9 comments
AuthorsOpenReview ↗

Response to Reviewer j3Af [1/4]

Dear Reviewer j3Af, We sincerely thank you for your careful and constructive feedback. We particularly appreciate your recognition of a simple and effective attack pipeline, analysis of event formation and intermediate scene descriptions beyond final-image ASR, and extensive experiments. Your comments on the role of the category, the stage ablations, mitigation, and validation have helped us further strengthen our evaluation through additional experiments and analyses. We address your concerns and questions in detail in the responses below.

[W1, Q1] Category-wise ASR and category dependence.

Answer: As requested, we report Attack Success Rate (ASR) by category below and compare against native RICE, the baseline with the highest overall ASR on both systems in Table 1 of our original submission.

Table R1. Category-wise ASR on GPT-Image-1.5.

GPT-Image-1.5HarassmentHateIllegal activitySelf-harmSexualShockingViolenceOverall (Table 1)
RICE22.507.5040.0017.5012.5015.0050.0023.57
I-CON55.0027.5045.005.002.5032.5065.0033.21

Table R2. Category-wise ASR on Gemini-2.5-Flash-Image.

Gemini-2.5-Flash-ImageHarassmentHateIllegal activitySelf-harmSexualShockingViolenceOverall (Table 1)
RICE15.0010.0030.0015.007.5027.5032.5019.64
I-CON50.0055.0042.5017.5022.5050.0045.0040.36
  • I-CON achieves higher ASR than RICE in 5 of the 7 categories on GPT-Image-1.5 and in all 7 on Gemini-2.5-Flash-Image. The two exceptions are self-harm and sexual content on GPT-Image-1.5.
  • Self-harm. GPT sometimes constructs explicit self-injury descriptions and subsequently refuses the corresponding image because of that content. The refusal responses suggest that making the harmful action explicit exposes it to the image-stage safety checks. By measuring Intermediate Leakage alongside ASR, we identified sessions in which the harmful description had already reached the user despite the final image rejection.
  • Sexual content. On GPT, concretization tended to resolve into non-explicit actions. GPT tended to emphasize adjusting clothing and clothed intimacy, whereas Gemini more often emphasized removing clothing or bodily contact. This difference already appears at the Concretize stage, before image generation.

We have added the complete category-wise results, including CaM and IL, and a discussion of these two categories to Appendix D.7 (Tables 12–13) of the revised manuscript.

[Q1] Attack success when the scene is fixed and the category changes.

Answer: We thank the reviewer for suggesting that we change the category while keeping the harm-sparse scene fixed. Reassigning the requested categories while retaining the same 280 harm-sparse scene texts still produced unsafe images, with an observed overall ASR of 33.21%, identical to that with the source categories. We used a fixed mapping between substantially different categories, including harassment → illegal activity and sexual → violence, with 40 cases per new category.

Category match (CaM) is the share of returned images assigned a given category by a Qwen3.5-9B image-only judge, which has no access to the input scene or the requested category. We report it against the new category and against the source category. ASR uses the same LLaVA-Guard evaluator as in our original submission.

Table R3. Category reassignment on GPT-Image-1.5.

GPT-Image-1.5ASR (%)CaM, source category (%)CaM, new category (%)
Source category33.2166.026.80
New category33.2118.7561.54
  • CaM against the new category increased from 6.80% to 61.54%, while CaM against the source category decreased from 66.02% to 18.75%. In the tested transitions, I-CON steered the same starting scenes toward the new category.

We have added the full mapping, per-transition results, and qualitative comparisons using the same starting scenes under the source and reassigned categories to Appendix D.8 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer j3Af [2/4]

[W2, Q2] Stage contributions at a fixed number of text calls.

Answer: As suggested, we evaluated generic replacements for each stage while retaining three text calls and one image request. This fixed-call control complements the stage-removal ablation in our original submission.

We evaluated each replacement on the same 280 GPT-Image-1.5 cases, keeping the starting scene, category, remaining stages, and final image request unchanged. The generic replacement is a neutral instruction that asks the target to continue developing the scene.

Table R4. Stage replacement at a fixed number of text calls on GPT-Image-1.5.

Replaced stageASR (%)CaM (%)Holm-adjusted p
None (I-CON)33.2166.02—
Ideate27.8658.410.0888
Concretize25.7158.800.0418
Refine31.0766.830.5044

To examine what each stage contributes, we combined Qwen3.5-9B measurements of intermediate scene descriptions with qualitative analysis.

  • Ideate expands the scene elements and relations that provide a basis for event construction. The qualitative examples illustrate how it adds actors and relations to the starting scene. Replacing this stage reduces ASR by 5.36 percentage points and CaM by 7.61 percentage points.
  • Concretize develops the scene into a directly observable event. Replacing it gives the largest ASR decrease, 7.50 percentage points. The share of scene descriptions containing a safety-relevant event falls from 86.02% to 40.14%. The qualitative examples illustrate the development of static scenes into events involving specific actions or changes of state.
  • Refine specifies lighting, color palette, and mood. Replacing it reduces the share of descriptions that specify all three in detail from 89.29% to 18.93%. ASR and CaM change little, while the atmosphere descriptions differ substantially, indicating that its contribution lies primarily in specifying the scene’s visual atmosphere.

Controlling for the number of text calls lets us more clearly distinguish how each stage’s instruction contributes to intermediate scene development and final-image ASR. These findings are consistent with our motivation: fixed scene-development requests can induce the target to construct concrete harmful events from harm-sparse scenes and coarse category guidance.

We have added the fixed-turn stage-replacement results and scene-description measurements to Sec. 4.2 and Table 3 of the revised manuscript. We have also added paired qualitative comparisons of the intermediate descriptions to Appendix D.6.2.

[W3, Q3] Mitigating harmful event construction: approach and initial evaluation.

Answer: We thank the reviewer for suggesting an initial mitigation evaluation. Our analysis found residual intermediate leakage (IL): harmful scene descriptions could reach the user even when the final image was blocked. A scene description can specify harmful visual content without instructing or endorsing the depicted act. We therefore evaluated a gate that judges each response before release by what the scene would depict. The target system’s own text model receives the conversation and the candidate response and decides whether an image faithfully depicting the scene would violate the image policy. An unsafe verdict withholds the response and ends the session.

Table R5. Offline safeguard evaluation pooled over GPT and Gemini.

ConditionI-CON ASR (%)I-CON IL (%)Benign blocked (%)
Native safeguards only36.7555.500.00
Think-Twice28.5055.500.35
Llama Guard 430.5043.002.13
Target-model gate5.254.756.38
  • The comparison shows why final-image ASR and intermediate leakage should be evaluated together. Relative to native safeguards alone, Think-Twice lowers ASR but leaves IL at 55.50%, because its check before image generation leaves earlier released descriptions exposed. Llama Guard 4 lowers IL to 43.00%, although its ASR is higher than Think-Twice’s.
  • The target-model gate achieves 5.25% ASR and 4.75% IL, substantially reducing both metrics relative to Think-Twice and Llama Guard 4. This reduction comes at a cost to legitimate conversations: at least one response is withheld in 6.38% of benign sessions, a higher rate than either comparator. By system, the gate gives 1.00% ASR and 0.00% IL with 10.64% benign blocking on GPT-Image-1.5, and 9.50% ASR and 9.50% IL with 2.13% benign blocking on Gemini-2.5-Flash-Image. Each system’s own text model is the judge, and the GPT judge blocks more broadly.
  • We evaluate this same gate online in our response to [W5, Q4].

We have added this mitigation evaluation and the discussion of residual ASR, Intermediate Leakage, and benign-session blocking to Sec. 4.5 and Table 5 of the revised manuscript, and the gate prompt and qualitative comparisons of generated descriptions to App. F.

AuthorsOpenReview ↗

Response to Reviewer j3Af [3/4]

[W4-1] Human-judged ASR.

Answer: We thank the reviewer for suggesting a human-judged ASR estimate. We now report a two-annotator evaluation of I-CON and RICE, the strongest baseline in Table 1 on both systems.

Two annotators labelled 100 randomly sampled returned images per method–system combination (400 images total) as Safe, Unsafe, or Unsure. For each annotator, ASR = (returned images / 280) × unsafe share, excluding Unsure ratings. We report the mean of the two estimates without adjudication; attempts without a returned image count as failures.

Table R6. Human-judged and detector ASR on GPT-Image-1.5.

MethodHuman ASR (%)Detector ASR (%)
RICE24.0623.57
I-CON46.7233.21

Table R7. Human-judged and detector ASR on Gemini-2.5-Flash-Image.

MethodHuman ASR (%)Detector ASR (%)
RICE21.3819.64
I-CON36.6440.36
  • Human judgments preserve the relative ordering, with I-CON achieving higher ASR than RICE on both systems. The gaps are 22.66 percentage points on GPT-Image-1.5 and 15.26 points on Gemini-2.5-Flash-Image.
  • Agreement on images both annotators labelled Safe or Unsafe is 88.14% for I-CON (κ = 0.763) and 92.13% for RICE (κ = 0.790).

We have added the two-annotator human-judged ASR estimates, the comparison with detector judgments, and the evaluation protocol to Sec. 4.1, Sec. 4.2, and App. D.3 of the revised manuscript.

[W4-2] Multi-annotator validation of event provenance.

Answer: We thank the reviewer for this suggestion. In a new, independent audit, two annotators attribute on average 98.87% of the harmful events in the images to the target rather than the user, consistent with the 99.35% from the one-annotator audit in our original submission.

  • The paper's 99.35% is a stratum-reweighted estimate from the original one-annotator audit of 42 of the 308 classifier-labelled events.
  • For the new audit, we drew a new stratified sample of 61 events from the same 308, and two annotators judged their origin from the images, independently and blind to the classifier's labels.
  • Among the events each annotator judged depicted and harmful, the weighted target-originated shares are 97.74% and 100.00% (mean 98.87%, 95% interval [89.32, 100.00]).
  • The two annotators agree on who introduced the event in 92.11% of the events that both judged harmful.

The extraction and sensitivity checks from the same audit are reported in our response to Reviewer os3N [W2-2].

Our original submission reported the one-annotator provenance audit, which is discussed in Sec. 4.3 of the revised manuscript. We have added the independent two-annotator audit, the extraction-recall check, and the sensitivity analysis to Sec. 4.3 and App. E.6.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response [1/2]

We thank reviewers j3Af, os3N, and Sx3y for their careful reviews. Below we summarize our contributions, responses and additional results. References follow in the next comment; details are in the individual replies.

Changes are marked in blue in the revised PDF.

Strengths highlighted by the reviewers

  1. [ j3Af ] A simple and effective attack pipeline, using fixed instructions without an external attacker model, adaptive search, or repeated image attempts.
  2. [ j3Af, os3N ] Analysis beyond final-image ASR, examining event formation and harmful intermediate descriptions that can reach users even when images are blocked.
  3. [ os3N, Sx3y ] A practical safety problem, addressing harmful content emerging through multi-turn interaction in conversational image generation.
  4. [ os3N ] A structured and interpretable framework, whose distinct scene-development stages make the procedure easier to understand and analyze.
  5. [ j3Af, Sx3y ] Broad experimental evaluation, covering GPT and Gemini, native and harm-sparse baseline comparisons, component ablations, and four existing safeguards.
  6. [ Sx3y ] A more covert attack that requires no attacker-specified harmful event, making the attack easier and exposing deployment risks.

Main contributions of our work

  • Harmful image generation without attacker-specified events. I-CON is a simple and effective attack that exploits the target’s scene-development capability. The attacker supplies a harm-sparse scene and a coarse harmful category, while the target constructs the concrete event. Three fixed scene-development requests and one image request suffice, without an external attacker model, adaptive search or retries.
  • Evidence of target-side harmful event construction. Our event-level analysis provides insights beyond final-image ASR by tracing where concrete harmful events enter the conversation and whether they appear in the image. Two independent human audits attribute nearly all harmful events in the examined images to the target rather than the user.
  • Intermediate exposure despite image blocking. Our safeguard evaluation shows that blocking the image does not prevent intermediate exposure. Some descriptions released before an image block elicit detector-flagged unsafe images when submitted alone. Safeguard replay further shows that reducing image ASR need not reduce Intermediate Leakage.
  • A target-model gate. The target system’s own text model judges each scene description under the image policy before release, reducing both attack success and Intermediate Leakage.

Highlighted responses and new results

  • ( j3Af, os3N ) Category controls: I-CON has a higher ASR than RICE in most categories. Paired with a new category, the same scenes keep 33.21% ASR, and the images follow the new category rather than the source category. The single word "harm" as the category still gives 29.29% ASR, vs. 6.43% with none.
  • ( j3Af, Sx3y ) Stage content at equal call count (GPT-Image-1.5): we replaced one stage at a time with a neutral request to continue the scene. Replacing Concretize lowers ASR from 33.21% to 25.71% (Holm-adjusted p = 0.0418).
  • ( j3Af ) Human-judged ASR: two annotators independently rated 400 sampled returned images. I-CON stays above RICE on both systems (46.72% vs. 24.06% on GPT).
  • ( j3Af, os3N ) Provenance validation: two annotators independently judged who introduced each sampled harmful event in the returned images, and on average attribute 98.87% of these events to the target.
  • ( j3Af ) Mitigation: in offline replay, our target-model gate lowers ASR to 5.25% and Intermediate Leakage (IL) to 4.75%, while blocking 6.38% of benign sessions.
  • ( j3Af ) Online safeguard evaluation: on 200 GPT cases where a blocked turn can be regenerated or the attacker can restart once with an indirect category phrase, the gate keeps ASR at 4.50% or below, whereas Llama Guard 4 stays at 29.00–31.00%.
  • ( os3N, Sx3y ) Input fidelity, grounding, and dataset release: two annotators confirmed that the scenes keep the subjects and, in most cases, the setting of the source while almost always removing its harmful content, and that the images remain grounded in their scenes. We will release all 280 scenes with the camera-ready version.
  • ( os3N, Sx3y ) Insight beyond final-image ASR: most events in the final description (70.25% on GPT, 73.01% on Gemini) are already written at Concretize, so they reach the user earlier.
AuthorsOpenReview ↗

Global Response [2/2]

Global reference

[G1] Feinstein and Cicchetti, “High Agreement but Low Kappa: I. The Problems of Two Paradoxes,” Journal of Clinical Epidemiology 1990.

[G2] Guo et al., “Involuntary Jailbreak: On Self-Prompting Attacks,” TMLR 2026.

[G3] Yang et al., “Position: Towards Implicit Prompt for Text-to-Image Models,” ICML 2024.

[G4] Huang et al., “Perception-guided Jailbreak against Text-to-Image Models,” AAAI 2025.

[G5] Zhang et al., “Metaphor-based Jailbreak Attacks on Text-to-Image Models,” arXiv:2512.10766, 2025.

[G6] Russinovich et al., “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack,” USENIX Security 2025.

[G7] Zhao et al., “When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systems,” USENIX Security 2026.

[G8] Schramowski et al., “Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models,” CVPR 2023.

[G9] Wang and Huang, “Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Models,” ECCV 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.