- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why vision-language models (VLMs) may comply with harmful instructions presented as images while refusing semantically equivalent instructions provided as text. The authors propose Safety Information Flow Tracing (SIFT), which analyzes attention messages from modality-specific source tokens to the assistant query state across decoder layers. By separately measuring semantic retention and refusal-related information, the method identifies a Safety Translation Bottleneck (STB), where visual messages retain decodable harmful semantics but exhibit weakened refusal-related signals. The authors further evaluate this mechanism using grouped edge knockout and bidirectional message swapping, and introduce SIFT-BRIDGE, a low-rank message translator for detecting and repairing cross-modal safety failures. Experiments are conducted on four VLMs, with additional evaluations on several external multimodal safety benchmarks.
Critical strengths
I think the paper goes beyond documenting jailbreak vulnerabilities and asks where harmful semantics and refusal signals diverge inside VLMs. The paired text-image design keeps the instruction fixed while swapping modality, which controls for several confounds. Calibrating refusal representations on harmful-only examples is a nice touch that separates refusal behavior from general harmfulness recognition.
Beyond probing, the paper includes grouped edge knockout, bidirectional message swapping, and intervention controls (Sec 5.3, Fig 4, Table 2). The 64% refusal rescue rate under matched swapping is concrete evidence that the localized pathway affects final behavior. Consistent localization across four VLMs and solid detection on held-out categories and external benchmarks further support the contribution.
Critical weaknesses
W1. Insufficient evidence for the proposed safety-translation mechanism.
Risk-category decoding (Section 3.3, Figure 2) does not necessarily demonstrate understanding of specific harmful instructions. Moreover, the refusal-control subspace (Eq. 3) may capture differences in jailbreak framing rather than refusal control alone. Although message swapping affects refusal rates, it does not establish corresponding changes in harmful compliance (Table 2). These limitations weaken the claimed mechanistic interpretation.
W2. Unclear source of SIFT-BRIDGE's performance gains.
The detection score in Eqs. (6)–(7) is mathematically equivalent to a linear readout of the visual message. It remains unclear whether the gains come from the proposed translation mechanism or simply from localized representations and paired supervision. A direct linear probe using the same STB message and matched supervision would be an important comparison.
Questions for authors
Q1. Evaluation split and STB localization.
The paper uses a 13-class semantic probe for STB selection (Section 3.3), while holding out entire risk categories (Section 4). How is this probe trained and used without exposing test categories during localization? Please clarify the category splits used for semantic probing and layer selection.
Q2. Attribution of SIFT-BRIDGE's improvements.
Equation (6) defines SIFT-BRIDGE as a reduced-rank ridge map, yet Table 7 reports a substantially weaker paired ridge baseline (0.82 vs. 0.93 AUROC). What precisely distinguishes these methods? How would a direct linear probe on the same STB message perform under matched supervision and tuning budgets?
Q3. Semantic retention and refusal-control specificity.
Can the authors provide stronger evidence of instruction-level semantic retention beyond risk-category classification? Additionally, does message swapping change actual harmful compliance or StrongREJECT severity rather than only refusal rates? These results would help substantiate the proposed safety-translation mechanism.