- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an image-only white-box jailbreak for MLLMs. Using matched harmful/harmless pairs, it analyzes four representation positions and picks the post-instruction token. It then selects a refusal direction (rejected minus accepted mean) through activation steering, and finally optimizes an ℓ∞ = 8/255 image perturbation that pushes the representation away from this direction. ASR is 95.65/93.04/98.69 on Llama-3.2-11B, Qwen2.5-VL-7B and MiniGPT-4-13B, slightly above JailBound with joint image+text perturbation.
Critical strengths
- The position analysis with matched pairs is clean. MiniGPT-4 shows no behaviour-aligned pattern and also refuses much later (median first refusal token 133.5 vs. 21/3), which nicely links representation to behaviour.
- The attack is simple, and the direction still works on held-out categories (Table 2).
Critical weaknesses
- ASR seems to be keyword-based. App. A.2 says the refusal templates are used "to evaluate attack success", while §6.4 mentions JailBound's judge template. If it is keywords, off-topic or misread outputs count as success; No-Attack ASR is already 73% on Llama. The refusal direction is also built from the same keyword labels used for scoring. In typo images the harmful phrase is inside the image, so the perturbation may simply stop the model from reading it. A judge-based harmfulness metric is needed.
- The claimed contribution is the objective (internal state vs. output likelihood), but no output-likelihood image attack is run under the same ε, steps and evaluator. The gap between JailBound (I) at 54.81 and 93.04 on Qwen is too large to accept without such a control. The ablation shows L_perp and L_reg add less than 1 point, so the method is close to existing refusal-direction attacks [1, 2], and these should be discussed.
[1] Refusal in Language Models Is Mediated by a Single Direction. NIPS, 2024. [2] Uncovering Safety Risks of Large Language Models through Concept Activation Vector. NIPS, 2024.
Questions for authors
- The "where" and "how" are chosen from similarity analysis and activation addition, but the attack itself is never run with the alternatives. What ASR do you get when the same attack targets the question-last token or the image tokens, or uses the input-based direction?
- Is the matched multimodal pair construction necessary? Does a refusal direction computed from text-only harmful/harmless prompts, as in [1], give similar ASR?
- Since the attack moves the post-instruction state along a known direction, a defender could simply monitor the projection onto d_ref. How easily are your adversarial images detected this way, and how much ASR is lost if the attack must stay below such a monitor?
- Does the perturbation transfer across models, or work as a universal perturbation?
[1] Refusal in Language Models Is Mediated by a Single Direction. NIPS, 2024.