- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper argues that effective refusal in VLMs need be neither text-conditioned nor high-dimensional. The authors observe, via singular-value spectra across three VLM families, that visual-harmfulness differences and refusal gradients are both concentrated in low-dimensional subspaces, with refusal more concentrated than harmfulness. Building on this, they construct two fixed rank-16 bases per model: a harmfulness basis Q_rep from SVD of normalized harmful-minus-safe last-token hidden-state differences, and a refusal basis Q_grad from SVD of refusal-margin gradients. A layer is chosen to maximize visual-minus-textual alignment. They then learn only a 16×16 matrix and a bias that maps harmfulness coordinates to refusal coefficients, applied as an input-dependent, rank-constrained hidden-state update at inference. On Qwen2-VL, InternVL3, and LLaVA-1.5, the method reduces MM-SafetyBench ASR by 5.0–18.4 points and HoliSafe unsafe ASR by 10.7–19.2 points, keeps MME nearly unchanged, and largely preserves MOSSBench benign behavior, at much lower compute than the compared defenses. A separate analysis reports a trade-off in which text-conditioned refusal is safer while visual conditioning better preserves benign behavior.
Critical strengths
S1. A cheap, well-specified intervention with a clear efficiency story. Learning a single 16×16 map (272 parameters) over two fixed SVD bases, with a frozen backbone and no auxiliary model or extra VLM pass, is genuinely lightweight, and Table 3 documents the compute advantage over seven baselines fairly.
S2. A clean conceptual separation that is novel in framing. Decomposing control into a harmfulness space (how the input departs from benign) and a refusal space (which directions increase refusal), and learning only the coupling between them, is an interpretable and well-motivated formulation. Driving refusal from a visually conditioned space rather than reusing textual safety representations is a genuinely different design point from most prior VLM defenses.
S3. A useful secondary contribution. The visual-vs-textual refusal-space comparison (Table 4) isolates a safety-versus-benign-behavior trade-off by varying only the conditioning of Q_grad, which is an informative finding in its own right and strengthens the motivation for the visual conditioning choice.
Critical weaknesses
The method is cheap and cleanly designed, and I like the harmfulness-space / refusal-space separation, but the paper is framed as a safety advance that its own tables don’t support (ETA and OH-Defense are safer in Table 2), the absolute ASR after defense is still very high, and the central low-dimensional claim is never tested against the orthogonal-complement attack the paper itself admits it invites. Details as follows:
W1. The safety narrative is not supported by the tables. The paper is framed as a safety defense, but by its own Table 2 it is never the safest method: ETA has lower MM-SafetyBench ASR on all three backbones, OH-Defense is far lower, and on HoliSafe the method is only second-best. The real contribution is a trade-off, not a safety gain.
W2. Absolute safety remains poor. Even after the reported reductions, LLaVA still has 77.8% HoliSafe ASR and Qwen2-VL 50.8% MM-SafetyBench ASR, so most attacks still succeed. The paper leads with percentage-point drops without arguing the practical value at this absolute level.
W3. No adaptive attack, which the method most invites. Appendix C concedes that any perturbation in the orthogonal complement of Q_rep bypasses the rank-16 update, then leaves evaluating this to future work. For a defense whose thesis is that a tiny subspace suffices, this is the key missing experiment.
W4. The low-dimensional claim is under-evidenced. Figure 1 compares only against an isotropic Gaussian, but hidden states are strongly anisotropic with outlier dimensions, so beating that null is weak support. A PCA-spectrum or shuffle control would be the appropriate baseline.
W5. Generalization has a clear counterexample. Qwen3-VL barely changes (MM-SafetyBench 41.5→40.1, HoliSafe 99.8→99.8), showing the single next-token rank-16 space can fail entirely when refusal is less globally shared.
W6. Over-refusal is unstable and model-dependent. MOSSBench refusal rises from 5.3% to 11.7% on Qwen, 3.7% to 19.7% on Qwen2.5-VL, and 10.3% to 22.0% on LLaVA-OneVision, and the method is not trained to control it.
Questions for authors
Please refer to weaknesses.
4 comments
Response to Reviewer 2K4N (2/2)
W4. The low-dimensional claim is under-evidenced.
The reviewer is right that low-dimensional spectral concentration is not specific to harmfulness and refusal. Following the reviewer’s suggestion, we examined the spectra of ordinary model features and gradients and found that they also show strong low-dimensional structure. This makes the original isotropic Gaussian baseline in Figure 1 too weak for arguing that safety-related representations are uniquely low-dimensional, so we removed that baseline and revised the corresponding discussion.
This does not change the motivation of our method. What we need is simply that harmfulness and refusal signals can be represented in compact subspaces, so that control can be carried out efficiently in a low-rank space. We have revised the main text to make this scope explicit and no longer present low-dimensionality as a property unique to harmfulness or refusal.
W5. Qwen3-VL is a counterexample to generalization.
Qwen3-VL is the only clear negative case among the nine VLMs we evaluated, while the other eight models show consistent safety improvements; our new Qwen3.5-9B and text-only LLM results further suggest that this is a model-specific boundary case rather than a general failure to transfer. We have therefore kept the Qwen3-VL result explicit and clarified this limitation in the revised paper.
The additional results support this interpretation across both newer VLMs and text-only LLMs. In the updated Table 7, Qwen3.5-9B improves from 47.2% to 42.1% on MM-SafetyBench and from 52.0% to 46.0% on HoliSafe, while MME remains nearly unchanged (2393.4 → 2384.7) and MOSSBench changes only slightly (6.7% → 7.7%). We also extended the method to four text-only LLMs (Table 9), where HarmBench and SorryBench ASR decrease on all four models and JailbreakBench ASR either decreases or remains unchanged. These broader results make Qwen3-VL the clear exception in our current evaluation rather than the dominant generalization pattern.
W6. Over-refusal is unstable and model-dependent.
Over-refusal is indeed less stable across architectures, but the overall benign refusal rates remain competitive with the compared safety methods. On the three main backbones, our MOSSBench FRR is 11.7%, 5.7%, and 11.3%, substantially lower than AutoSteer (22.7%, 9.3%, and 20.3%) and lower than NullSteer on two of the three models (10.7%, 33.0%, and 51.3%) (Table 2).
The model-dependent increases highlighted by the reviewer are real and deserve explicit discussion. MOSSBench FRR rises from 5.3% to 11.7% on Qwen2-VL, from 3.7% to 19.7% on Qwen2.5-VL, and from 10.3% to 22.0% on LLaVA-OneVision (Tables 1 and 7), showing that benign refusal calibration is not equally stable across architectures.
This level of over-refusal control is encouraging because MOSSBench FRR is not directly optimized by our training objective. The preservation loss encourages the modified model to stay close to the original behavior on benign inputs, but it does not explicitly optimize an over-refusal metric. We have clarified this limitation and identify direct benign-refusal calibration as an important direction for making the trade-off more consistent across models.
Thank you for the detailed response and the revised manuscript. The reframing around the safety–utility–efficiency trade-off, the removal of the isotropic baseline, the additional backbones, and the Qwen3-VL analysis make the claims better aligned with the evidence, and I have raised my soundness score accordingly. However, the main concerns were addressed by narrowing the claims rather than by new evidence that overturns them: the method’s absolute safety remains mid-range among the compared defenses, the low-dimensional observation is no longer specific to harmfulness or refusal, and the adaptive-attack results in Appendix F lack an undefended reference, so they do not yet establish robustness. I therefore keep my overall rating.
Dear Reviewer 2K4N,
Thank you again for your careful and insightful review. In the rebuttal, we directly addressed the key issues you identified: most importantly, we evaluated the orthogonal-complement attack in both hidden-state and realizable pixel space, added stronger controls for the low-dimensionality analysis, revised the framing of the safety–utility trade-off, and added results on newer VLM backbones.
We believe these additions materially strengthen the support for our core claims and make the scope and positioning of the paper more precise. We would greatly appreciate it if you could revisit our response and revised manuscript, and reconsider your overall assessment and rating based on the revised framing and strengthened support for the paper’s central claims.
Thank you very much for your time.
Response to Reviewer 2K4N (1/2)
We thank the reviewer for the thoughtful and direct feedback. The comments helped us sharpen the paper’s central message: safety should be evaluated together with utility and inference cost, since a defense that substantially degrades model capability or adds heavy inference overhead has limited practical value. Our method offers a favorable combination of meaningful safety gains, near-vanilla utility, and low additional inference cost (Tables 1–3). We have revised the manuscript to foreground this safety–utility–efficiency trade-off, with the major changes highlighted in blue.
W1. The safety narrative is not supported by the tables.
The central contribution is a strong safety–utility–efficiency trade-off, and we have revised the Introduction and contribution statement to foreground this point. The two methods with clearly lower ASR, OH-Defense and ETA, incur substantial costs in utility and inference efficiency, respectively.
OH-Defense achieves stronger absolute safety (Table 2), but its MME scores drop to 1598.6, 1820.2, and 1378.2, compared with 2317.4, 2385.0, and 1664.0 for ours on Qwen2-VL, InternVL3, and LLaVA-1.5 (Table 1). ETA also achieves lower ASR (Table 2), but requires auxiliary ArmoRM and CLIP models, additional scoring, and conditional Best-of-5 generation (Table 3). Our method therefore provides a favorable operating point with strong safety gains, near-vanilla utility, and minimal inference overhead.
W2. Absolute safety remains poor.
Our goal is to achieve a strong safety–utility–efficiency trade-off, and the reported operating point is chosen with this overall objective in mind. We acknowledge that the remaining ASR is still high in absolute terms, and we have revised the paper to make these absolute values and the corresponding trade-off clearer.
Even when safety is considered on its own, our method outperforms most compared defenses. On Qwen2-VL MM-SafetyBench, most methods remain above 55% ASR, while ours reaches 50.8%; on LLaVA HoliSafe, most remain above 80%, while ours reaches 77.8% (Table 2). The two main methods with substantially lower ASR are OH-Defense and ETA: OH-Defense incurs a large utility drop (Table 1), while ETA requires substantially higher inference cost through auxiliary models and additional generation (Table 3).
If the sole objective is to further reduce ASR, the intervention strength α provides a direct way to do so. As shown in Figure 4, increasing α can provide additional safety gains. We use α=1 in the main results because it gives a favorable balance among safety, general utility, and benign behavior.
W3. No adaptive attack against the orthogonal complement.
This is a valid concern under a full white-box threat model, although such an attacker must directly access and manipulate internal hidden states, which is substantially stronger than the input-space attacks available in most practical settings. We therefore evaluated the orthogonal-complement attack at both levels and added the results and discussion to Appendix F.
We selected 50 examples that were originally answered safely under our defense and applied the following adaptive attacks:
Under direct hidden-state manipulation, the reviewer’s proposed bypass is effective. Restricting the perturbation entirely to Qrep⊥ still makes 23/50 previously safe examples unsafe, showing that the low-rank intervention does not provide robustness to an attacker with direct control over internal representations.
The attack becomes substantially less effective when constrained to realizable pixel-space perturbations. At ϵ=8, standard PGD reaches 28% ASR, while adding the orthogonal-complement objective gives 22% at λ=1 and 18% at λ=10. Across the broader sweep, none of the orthogonally constrained pixel-space attacks outperforms standard PGD.
Overall, the experiment identifies a genuine white-box limitation, while also showing a substantial gap between direct latent manipulation and attacks that must operate through the input image. We now discuss this distinction explicitly in Appendix F and include adaptive robustness as an important direction for future work.