acceptodds
Under review as a conference paper at ICLR 2027

Safety Lost in Translation: Where Harmful Intent Disappears in Vision-Language Models

Abstract

Vision-language models (VLMs) can refuse a harmful instruction in text yet comply when the same intent is delivered through pixels. Existing work establishes this safety gap across modalities, but does not isolate the computation that separates successful visual understanding from failed refusal. We introduce SIFT (Safety Information Flow Tracing), a paired analysis of the attention message sent from each modality into the shared assistant query state. To separate harmful semantics from refusal itself, refusal control is calibrated on pairs of harmful text prompts with the same underlying goal, one refused and one complied with. We localize the Safety Translation Bottleneck (STB) using a continuous refusal logit gap, with CKA, CCA, and scalar magnitude checks as robustness controls. At the STB, visual messages retain decodable harmful semantics while losing refusal control strength. Grouped edge knockout from source tokens to the query independently peaks in the same depth interval, and bidirectional message swaps on held out examples change the final response rather than only an internal score. The same mechanism motivates SIFT-BRIDGE, a rank 8 translator that repairs only the localized message channel. On a paired bank of 1,680 MM-SafetyBench intents, we obtain 0.93 AUROC for translation failure detection, compared with 0.83 for a paired multilayer probe with matched capacity. Independent replications cover LLaVA-1.5, LLaVA-1.6, Qwen2.5-VL, and InternVL3, while FigStep, VLSBench, JailBreakV-28K, and UnsafeConcepts test transfer beyond rendered text.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dGbZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why vision-language models (VLMs) may comply with harmful instructions presented as images while refusing semantically equivalent instructions provided as text. The authors propose Safety Information Flow Tracing (SIFT), which analyzes attention messages from modality-specific source tokens to the assistant query state across decoder layers. By separately measuring semantic retention and refusal-related information, the method identifies a Safety Translation Bottleneck (STB), where visual messages retain decodable harmful semantics but exhibit weakened refusal-related signals. The authors further evaluate this mechanism using grouped edge knockout and bidirectional message swapping, and introduce SIFT-BRIDGE, a low-rank message translator for detecting and repairing cross-modal safety failures. Experiments are conducted on four VLMs, with additional evaluations on several external multimodal safety benchmarks.

Critical strengths

I think the paper goes beyond documenting jailbreak vulnerabilities and asks where harmful semantics and refusal signals diverge inside VLMs. The paired text-image design keeps the instruction fixed while swapping modality, which controls for several confounds. Calibrating refusal representations on harmful-only examples is a nice touch that separates refusal behavior from general harmfulness recognition.

Beyond probing, the paper includes grouped edge knockout, bidirectional message swapping, and intervention controls (Sec 5.3, Fig 4, Table 2). The 64% refusal rescue rate under matched swapping is concrete evidence that the localized pathway affects final behavior. Consistent localization across four VLMs and solid detection on held-out categories and external benchmarks further support the contribution.

Critical weaknesses

W1. Insufficient evidence for the proposed safety-translation mechanism.

Risk-category decoding (Section 3.3, Figure 2) does not necessarily demonstrate understanding of specific harmful instructions. Moreover, the refusal-control subspace (Eq. 3) may capture differences in jailbreak framing rather than refusal control alone. Although message swapping affects refusal rates, it does not establish corresponding changes in harmful compliance (Table 2). These limitations weaken the claimed mechanistic interpretation.

W2. Unclear source of SIFT-BRIDGE's performance gains.

The detection score in Eqs. (6)–(7) is mathematically equivalent to a linear readout of the visual message. It remains unclear whether the gains come from the proposed translation mechanism or simply from localized representations and paired supervision. A direct linear probe using the same STB message and matched supervision would be an important comparison.

Questions for authors

Q1. Evaluation split and STB localization.

The paper uses a 13-class semantic probe for STB selection (Section 3.3), while holding out entire risk categories (Section 4). How is this probe trained and used without exposing test categories during localization? Please clarify the category splits used for semantic probing and layer selection.

Q2. Attribution of SIFT-BRIDGE's improvements.

Equation (6) defines SIFT-BRIDGE as a reduced-rank ridge map, yet Table 7 reports a substantially weaker paired ridge baseline (0.82 vs. 0.93 AUROC). What precisely distinguishes these methods? How would a direct linear probe on the same STB message perform under matched supervision and tuning budgets?

Q3. Semantic retention and refusal-control specificity.

Can the authors provide stronger evidence of instruction-level semantic retention beyond risk-category classification? Additionally, does message swapping change actual harmful compliance or StrongREJECT severity rather than only refusal rates? These results would help substantiate the proposed safety-translation mechanism.

Reviewer_nK2bOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes the known issue that VLMs are less robust against harmful text instructions embedded in the image channel compared to if those instructions were provided in the text channel. The paper aims to establish that the harmful content is represented similarly in the intermediate representations in the VLM independent of whether it comes from visual or text tokens, by showing that a probe trained on attention on text tokens can predict the harm category from attention on the image tokens. Additionally it builds a probe to isolate refusal from harmfulness using successful jailbreak and non-jailbreak versions of the same harmful input in the text channel. It then argues, that a layer in which the harmful category probe is predictive for image-channel instructions while the refusal probe is low on the image-channel instructions vs. the same text-channel instructions constitutes a ‘Safety Translation Bottleneck’ where the harmful content is still represented from the image channel, but doesn’t translate into refusal behavior. It offers empirical evidence that this divergence in representation can be localized in the attention messages in intermediate model layers. Finally, it trains a low-rank translator (SIFT Bridge) to predict the attention contribution from the text-channel instructions from the same image-channel instructions and uses it to detect when text and image-channel refusal behaviors disagree as well as repairing the image-channel refusal behavior by imputing the predicted text-channel behavior.

Critical strengths
  • The paper establishes that text-safety category information remains decodable from image-channel instructions while refusal behavior changes compared to the text channel and convincingly shows that this is decodable from differences in the attention messages coming from each channel.
Critical weaknesses
  • The paper focuses on the limited use case of harmful text instructions being embedded in the image channel and how it differs from the text channel. The simplest baseline that is missing is just running OCR and giving the OCR output directly to the text channel of the VLM (not the external classifier they reported). If we assume that OCR is near-perfect at this time, this would fix the issue by construction. So the practical value of the analysis seems very limited.
  • I feel like the strongest results are using SIFT bridge for jailbreak detection and refusal repair from image-channel but I am not convinced that the paper’s methodology is the best way of getting there. In my mind this should be treated like a standard prediction problem, with flexible classifiers from all kinds of internal representations and tuned on a validation set. This hasn’t been done and compared against (the existing baselines are capacity matched) so I am not convinced the whole STB narrative is the most practically useful way to think about this problem. Also, from this perspective it is very unclear why one should only predict the difference between text and image channel behavior? Why not directly try to improve the image channel against the gold label that we have from the data?
Questions for authors
  • I don’t understand why the detection results for the Reduced rank ridge baseline are 10 points worse than SIFT Bridge, because SIFT Bridge is nothing but a low-rank linear regression? What is the crucial difference that leads to the performance delta?
Reviewer_byf7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the reason why vision-language models can refuse a harmful instruction in text but comply when the same instruction is presented in an image. Inspired by the analysis, this paper proposed the SIFT framework that extracts attention messages from instruction bearing tokens to the last prompt position. The proposed method uses a semantic probe and a refusal readout to identify a Safety Translation Bottleneck (STB), where visual messages remain informative about risk categories while showing weaker refusal-related signals than their text counterparts. The authors then use held-out message swaps and knockouts to evaluate the influence of the method on model responses. The evaluation covers four VLMs, several heldout risk categories, and several external datasets. The reported results suggest that the selected messages matter for refusal and that the bridge can improve detection and reduce harmful responses.

Critical strengths
  1. The paper goes beyond measuring a visual language safety gap and tests whether the selected messages affect the outcome. I find the swap experiments are helpful and interesting, the paired text-to-visual swap achieves a 64% rescue rate, compared with 24% for a donor from a different risk category and 8% for a random donor. The reverse swap also reduces refusal. These results support that the selected messages take a causal role in refusal. Selecting the bottleneck separately from the intervention tests also makes this evidence more convincing.

  2. The method changes a selected attention rather than the full residual state, and the experimental results suggests that the approach can be very useful and effective. I can see value of this method that it trying making the intervention more specific while limiting its effect on normal behavior

Critical weaknesses
  1. The technical presentation seems a little incomplete in several places that are important to the paper. I can follow the overall idea, but I still have difficulty checking how several central results were obtained. The draft needs a revision of its technical description and reporting I think. I'll strongly recommend the authors rewrite part of the paper and update the figures.

  2. One of my main concern is on the claim that: "the harmful intent is preserved but fails to trigger refusal.". The probing experiment try predicting on 13 categories, while this does not show whether the message still contains the particular action being requested or the details needed to assess that request. The swap experiments are helpful, but they do not fully resolve this issue as well I think. For example, a paired text donor may restore information specific to the request, or it may mainly provide a stronger refusal signal. The random and different-category controls do not clearly separate these possibilities. A donor from the same risk category but with a different harmful action would be a more informative ablation study can consider. The current results make me believe that the selected messages can influence refusal, but Im less convinced that they establish the more specific explanation of safety being lost during translation.

Questions for authors

Some questions are mentioned in the weaknesses above. In addition, there are some more question which can be helpful for clarifying the paper if the authors can provide more information about them:

  1. The STB selection rule is a bit unclear to me: Section 3.3 defines a refusal gap for each layer, then selects a contiguous window that maximizes this gap. It is unclear how a window is scored, how its width is determined, and how the particular layer used by the bridge is chosen from that window. These details matter because the localization is a key of the method. It will be great if the author can plot the actual selection statistic and explain why the shaded region in Figure 3 differs from where the visible text-visual refusal gap appears largest?

  2. The evaluation includes several VLMs, but all are mainly in the 7B-8B range. Do the same bottleneck pattern and intervention effects hold at other sizes within the same family and in newer or different model families? Also, it would be helpful to compare the original text visual refusal gap, the swap effects, and the actual repair benefit, rather than detection AUROC alone.

  3. The diagnostic refusal subspace and the repair subspace are built from different contrasts. How does the repair update relate to the refusal signal used for diagnosis? For example, how much of the update lies in the diagnostic subspace, and does restricting the update to that subspace preserve its effect on final refusal, and also will the repaire hurt some capability of the model behavior on the general tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.