acceptodds
Under review as a conference paper at ICLR 2027

Jailbreaking Multimodal Large Language Models through Generative Visual Semantic Ciphers

Abstract

Multimodal large language models (MLLMs) can be induced to reconstruct harm- ful instructions from distributed visual and textual cues while failing to reliably apply safety constraints to the recovered intent. We investigate this failure mode through the Generative Visual Semantic Cipher Attack (GVSCA), a black-box framework that encodes instruction components as recognizable surrogate con- cepts within a single generated natural scene. Combined with attacker-provided semantic mappings and decoding guidance, the image induces the target model to reconstruct and respond to a harmful instruction that is not presented as a con- tiguous plaintext command. Across 11 MLLMs and 65 selected harmful instruc- tions spanning eight categories, GVSCA achieves average attack success rates of 69.76% with one query and 93.01% within ten queries, compared with 14.90% and 29.79%, respectively, for the strongest evaluated baseline. Matched-carrier experiments on three models show that replacing the generated images with tex- tual carriers reduces single-query success from 64.23% to at most 5.13%, with semantic mappings and reconstruction procedures held fixed. Joint image–text filtering with Llama Guard 4 still leaves a 55.69% single-query success rate. In a separate three-model evaluation, a system instruction requiring safety assessment of the reconstructed intent reduces single-query success from 70.39% to 20.00%, but incurs a 15.00% benign refusal rate. These findings highlight a safety gap between cross-modal semantic reconstruction and safety enforcement, motivating defenses that assess reconstructed instructions while preserving benign reasoning capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, jailbreak attacks, multimodal safety, cross-modal reasoning, semantic reconstruction, visual semantic cipher

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_U68LOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Generative Visual Semantic Cipher Attack. GVSCA is a black-box jailbreak against multimodal large language models (MLLMs). The attack works by 1) encoding harmful instructions as objects within a model-generated scene and 2) providing semantic mappings needed to reconstruct the original instruction. The model reconstructs the harmful instructions, but the process avoids triggering the safety mechanism. The paper also includes evaluation against defences and comparisons with a textual-only counterpart.

Critical strengths
  • The paper had a broad evaluation against 11 diverse MLLMs
  • The paper evaluates different defenses against the proposed GVSCA attacks
  • The paper is properly written and easy to follow
Critical weaknesses

The authors have addressed this concern in the revised manuscript. Therefore, I reassess the paper and increase my score to a Weak Accept.

Pre-rebuttal "Jailbreaking Vision-Language Models Through the Visual Modality" Azulay et al. ICML is an extremly closely related work that is neither cited nor compared against. The abstract of this paper states that "Generative Visual Semantic Cipher Attack (GVSCA) [is] a black-box framework that encodes instruction components as recognizable surrogate concepts within a single generated natural scene." Azulay et al.: 1) propose a visual cipher attack, and 2) explore the idea of "recognizable surrogate concepts" by replacing harmful concepts with benign ones and instructing the model to reconstruct the original meaning. This paper should be cited, and the differences should be properly discussed.

To be specific:

  • Both works encode harmful instructions as visual ciphers. Azulay et al. use an image with 1) a sequence of visual symbols and 2) a decoding legend. GVSCA maps instruction components to objects in a generated scene.
  • Both works use benign visual objects to represent harmful concepts.
  • Both works provide instructions for decoding the visual input.
  • Both works rely on the model reconstructing the harmful instruction. Azulay et al. ask the model to decode the visual cipher and execute the resulting instruction. GVSCA asks the model to identify objects, apply the mappings, and reconstruct the original instruction before responding.
  • Both works exploit the same weakness. Verbatim: GVSCA: models can "reconstruct harmful instructions from distributed visual and textual cues while failing to reliably apply safety constraints to the recovered intent." Azulay et al. study "whether visual context can reconstruct the latent harmful intent, thereby isolating cross-modal safety failures rather than purely textual ones."
  • Both works include ablation experiments comparing image attacks with text-only variants (which is in principle good, but the novelty of this submission cannot be claimed based on this ground).

The main difference seems to be that Azulay et al. use visual symbol sequences and object replacements, while GVSCA combines the visual objects into a single generated scene.

Questions for authors

To be clear, I am not claiming that the omission of Azulay et al. IMCL is intentional. However, the issue remains, and analysis of related work is one of the many responsibilities of researchers. Therefore:

  • Could the authors please clarify why Azulay et al. is omitted?
  • Could the authors clarify the novelty of the paper relative to Azulay et al.?
  • Could the authors experimentally compare GVSCA against the visual cipher attack of Azulay et al.?

I am in no way adversarial against this work. I am open to revising my score to acceptance depending on the authors' responses. I believe that the issue could be addressed in the rebuttal; however, with such closely related work, all 3 questions above need to be addressed to influence my score.

Reviewer_kuWvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GVSCA, a black-box jailbreak that represents instruction components as visual surrogate concepts within a generated scene. An attacker-provided codebook and decoding prompt guide the target model to reconstruct and answer the harmful instruction. Experiments on 11 models show high attack success, while additional studies examine textual carriers, input filtering and reconstruction-aware defense.

Critical strengths
  1. The same attack inputs work across 11 models without being adjusted for each model. The baselines use the same evaluation samples, and successful responses are checked by human reviewers.

  2. The authors compare images with text descriptions of the same concepts. They also test a defense and measure how often it refuses harmless mapping and puzzle tasks.

Critical weaknesses
  1. The comparison uses only the mirroring version of MML. It is unclear whether the advantage remains over its more closely related word replacement version.

  2. Reconstruction results are reported only for image inputs. The low success of textual carriers could reflect failed reconstruction or refusal after correct reconstruction.

  3. The experiments do not separate scene generation from visual presentation. A simple image containing the same objects would help show what scene generation adds.

Questions for authors
  1. Does the improvement remain against the word replacement version of MML?

  2. For each textual carrier, how often is reconstruction correct, and how often is a correctly reconstructed request refused?

  3. How does a generated scene compare with a simple visual arrangement using the same mappings and decoding prompt?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.