acceptodds
Withdrawn from ICLR 2027

Which Image Property Carries The Jailbreak? A Controlled Dissection Of Image-To-Text Jailbreaks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We ask which image-side property causes safety failure in multimodal image-to-text jailbreaks, and answer with a controlled dissection rather than another leaderboard entry. We hold the harmful instruction byte-identical, probe four published attack families across five victims (GPT-4o, Claude S.4.5, Qwen3-VL-8B, Gemma-3-4B, GLM-4.1V) on a 313-prompt benchmark slice, and score every response with an automated rubric judge; the paired ablations are sampled draws per prompt and the baseline matrix is single-draw, as declared in the method. The image is the carrier of the attack: a bare harmful query and a benign unrelated image sit at or near zero on every victim, while attaching an attack composite lifts success substantially. Three findings are robust across the panel. Per-tile density screening cannot separate attack tiles from size-matched benign distractors, so density-only screening is not a viable filter. The compositional ladders are either confounded by payload-text visibility or null, rather than rising with a clean image-side descriptor. On the headroom model with the most room to move, the topical relation between a generated scene and the harmful query raises refusal by about one tenth of the ASR scale (rubric-judged, three draws per prompt, single-victim), an observation that is bounded and descriptive rather than causal: it is attributed under the measured-descriptor rule to the relatedness manipulation, with image-text congruence unmeasured, and reproduced by a within-category harm-salience control on overlapping stimuli. The API and reasoning-augmented rows are floored or saturated, which bounds where an effect can be measured. The dissection is scoped to composite and typographic-layout families; every claim is conditional on our rubric judge, and we rely on signs and orderings rather than on magnitudes inside the roughly one-tenth cross-judge band.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, jailbreak attacks, vision-language model safety, image-to-text jailbreaks, controlled ablation, causal attribution, red-teaming, AI safety evaluation, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q8WtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission asks which image-side property is responsible for safety failure in image-to-text jailbreaks, and answers with a controlled dissection rather than another leaderboard comparison. Images are characterized along four axes — compositional complexity (tile/region count), information density (pixel entropy and JPEG@85 size), tile provenance (natural photographs vs. SD3.5-generated scenes), and semantic alignment to the query (whole-image CLIP similarity) — and four published attack families (FigStep, two MML variants, CS-DJ, VisCo) are placed in this space. The harmful query is held fixed across conditions, image-side descriptors are measured before any victim is queried, and a contrast is attributed to a descriptor only when that descriptor is the sole measured quantity that changes; otherwise it is labeled jointly attributed and excluded from headline claims. Experiments use the full StrongREJECT slice (n=313) against five victims (GPT-4o, Claude Sonnet 4.5, Qwen3-VL-8B, Gemma-3-4B, GLM-4.1V-9B-Thinking), with the StrongREJECT rubric judge, bootstrap confidence intervals, Holm-corrected paired McNemar tests, and a global multiplicity correction. The main findings are: (i) tile-count ladders are confounded by payload-text visibility, and a region-count ladder that holds payload visibility fixed is null on both tested victims; (ii) a padding/border difference between two renderings of the same CS-DJ composite moves ASR by 0.19 on Gemma-3-4B, larger than any designed axis; (iii) the only contrast surviving global correction is a ~0.12 ASR difference on Qwen3-VL-8B attributed to the query-relatedness of generated scenes; and (iv) the image is the carrier (bare queries and benign images sit near zero), while a per-tile entropy/JPEG density screen cannot separate attack tiles from benign tiles.

Critical strengths

S1. The experimental discipline is substantially above the norm for this literature. Holding the query string fixed, measuring entropy/compressibility/CLIP/provenance on every stimulus before any victim is queried, committing to a one-descriptor attribution rule, and using one judge, bootstrap CIs, paired McNemar tests, and a global multiplicity correction is exactly the kind of protocol the attack literature lacks. The authors also apply the rule to their own constructions and report the ladders it invalidates. This matters because it makes the negative results credible and gives other researchers a reusable template.

S2. Two concrete, reusable findings emerge. First, the monotone ASR rise across tile-count ladders is co-measured with payload legibility, and a region-count ladder that holds payload visibility fixed is null; this directly weakens the compositional-complexity explanation used by CS-DJ-style attacks. Second, the largest image-side effect in the paper is a padding/border construction difference (0.19 on Gemma-3-4B), which implies that reported ASR differences between two renderings of "the same" attack may be construction noise rather than mechanism. Both are immediately actionable for future attack and defense work.

Critical weaknesses

W1. The headline result is reported with the wrong sign, and its magnitude is within the paper's own noise band. The abstract states that the topical relation between the generated scene and the query "raises refusal by about one tenth of the ASR scale". Table 8 shows the opposite: relevant scenes reach 0.29 ASR while the provenance-matched generated-unrelated cell reaches 0.17, i.e., relatedness raises ASR (lowers refusal), Δ=−0.12 for relevant→gen-unrelated. Section 5.4 reports the within-category control as Δ=+0.089, while Appendix A.7 reports −0.089. Because this is the single contrast that survives the global correction, the direction of the paper's only positive claim must be unambiguous; as written, a reader cannot tell whether relatedness helps or hurts the attacker. In addition, an absolute effect of ~0.09–0.12 sits at the edge of the ~0.10 cross-judge band and of the 0.06–0.10 minimum detectable effect; the paper's own power analysis asks for n≈830 to confirm a 0.06 effect. The result is therefore a bounded, single-victim observation, and the paper's framing overstates how much of the title question it answers.

W2. The victim panel is outdated and the "byte-identical" control is under-specified. All five victims predate the submission by one to two model generations (GPT-4o, Gemma-3-4B, GLM-4.1V-9B-Thinking, Qwen3-VL-8B, Claude Sonnet 4.5), and the two API rows are floored or saturated by the authors' own account; current frontier models are absent, so it is unclear whether any conclusion transfers. Separately, the threat model defines the victim input as (I, t) with a "benign textual wrapper" t, but the paper never gives t, the prompt serialization, or how the query is routed into each attack family (typographic rendering vs. sub-query tiles vs. conversational scaffold). "Byte-identical" can therefore only mean that the same source query string is reused to build the stimuli; the actual victim input is re-rendered, downscaled to ≤768px, and JPEG-encoded, and differs across families by construction. Without the wrapper and template, the central control is not reproducible and cross-family ASR comparisons remain confounded. The related-work discussion also omits factorized/attribution studies such as MMJailBench (arXiv:2608.25490) and representation-shift analyses (arXiv:2603.17372), which makes it hard to see what is new relative to concurrent work.

Questions for authors
  1. The abstract reports 64.23% and 70.39%, whereas Tables 3 and 4 and Sections 4.3 and 4.5 report 64.21% and 70.36%. Which values are correct, and are any conclusions affected?
  2. In the matched-carrier experiment, how exactly are the textual carriers presented to the model, and is the decoding prompt adapted to the text conditions? The prompt refers to "the image" and to key objects/characters; if it is left unchanged, could the 5.13% result be explained by the mismatch rather than by modality?
  3. What is the aggregation in Table 23 — which models and which instructions are averaged — and why is Odysseus's ASR@10 at JPEG q=95 (33.33%) higher than its clean-condition average in Table 13 (16.08%)?
  4. How is "instruction-level concealment" operationalized? Concretely, what is concealed from which observer, and how would GVSCA differ from a text-side cipher with the same mapping table? Since the "no cipher" ablation drops to 0%, is the mechanism better described as instructed decoding than as reconstruction of hidden intent?
  5. How were the Rec and Dec labels assigned, and what was the inter-annotator agreement? Given that GPT-5.2 reconstructs correctly but is jailed only 70.77% of the time, what does the gap between reconstruction and jailbreak tell us about the safety failure being studied?
  6. Given the selection from 150 to 65 instructions, what is the estimated ASR over the full candidate pool (or at least over the 45 encoding-infeasible instances), and does the exclusion criterion correlate with harm category?
  7. Is there any evaluation of an adaptive attacker aware of the reconstruction-aware system instruction — e.g., instructions that disguise or defer the decoding step? How stable is the 20.00% residual ASR in that setting?
  8. Can you report confidence intervals or per-instruction variance for the main ASR numbers, and how sensitive is ASR to the choice of surrogate concept for a given semantic unit?
Reviewer_Gz14OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to investigate which image attributes, within existing image-to-text jailbreak attacks, actually impact model security. The authors analyze various established visual jailbreak methods by examining four dimensions: compositional complexity, information density, semantic alignment, and the origin of sub-images. By attempting to disentangle the effects of these distinct attributes, the authors evaluate the explanations provided by existing research regarding the mechanisms behind visual jailbreak attacks.

Critical strengths
  1. This paper reconsiders the explanation of existing visual jailbreak mechanisms. The authors point out that some previous studies attributed the increased success rate of attacks to the complexity of image composition, but these experiments actually incorporated the readability of harmful text.

  2. The experimental analysis in this paper is relatively comprehensive. The authors employed paired experiments, multiple comparison corrections, and cross-validation of different evaluation models, and proactively reported insignificant or confounded results.

Critical weaknesses
  1. This paper does not fully answer its core research questions. Although the authors proposed four dimensions of image attributes, most experiments failed to independently disentangle these attributes. The only relatively clear semantic relevance experiment also has other possible interpretations.

  2. Confirmatory evidence and generalization of the core findings remain limited. The main positive findings only received significant support on one model, and further experiments did not observe stable response relationships.

Questions for authors
  1. Could the authors vary only the number of tiles while keeping the harmful text readable and other image properties fixed, to directly test compositional complexity?

  2. Can the main finding be reproduced on an independent dataset with a pre-specified analysis plan? Why does the relatedness effect show no stable dose-response?

  3. Could the authors add the benign-generated unrelated and benign-paraphrase controls to separate semantic relatedness from refusals caused by image-text mismatch?

Reviewer_umbUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates what image properties impact the attack success rate (ASR) of multimodal LLMs. The experiments include controlled ablations of four attack families on five multimodal LLMs. The main finding of the paper is that the impact of image properties on ASR is substantial. Benign unrelated images together with harmful prompts are not successful, while specifically crafted composite adversarial images have high ASR.

Critical strengths
  • The core scientific question: what image properties impact the ASR of jailbreaks is interesting; if done and presented correctly, it could be valuable for the community
  • The paper evaluates on 5 diverse models: GPT-4o, SONNET 4.5, QWEN3-VL8B-INSTRUCT, GEMMA-3-4B-IT, GLM-4.1V-9B-THINKING.
Critical weaknesses
  • The writing seems to be AI-generated. This severely impacts the paper's clarity and makes it painful to read. Thus, the reader gains very little by interacting with the paper. The presentation of the results is where this is most problematic.
  • The experimental evaluation in the paper is quite limited. The scope is better suited to a workshop submission than to a conference like ICLR.
  • The evaluation is on small open-source models or on closed-source models that are quite old. Thus, the generalization of the findings is unclear.
Questions for authors

Suggestions for future work: Have you considered a human rewrite of the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.