acceptodds
Under review as a conference paper at ICLR 2027

From Visual Harmfulness to Refusal in Low Dimensions

Abstract

Safety defenses in VLMs often reuse textual refusal representations, but stronger control can come at the cost of benign and general multimodal behavior. We show that effective refusal need be neither text-conditioned nor high-dimensional. Visual harmfulness and refusal can both be captured in low-dimensional subspaces, with refusal concentrated in markedly fewer directions. We therefore propose a visually conditioned harmfulness-to-refusal mapping that connects the two low-dimensional spaces with only 272 trainable parameters. The mapping reduces overall attack success rate by 5.0–18.4 percentage points on MM-SafetyBench and 10.7–19.2 percentage points on HoliSafe, while leaving MME performance nearly unchanged and largely preserving benign behavior on MOSSBench. Comparing the two refusal spaces reveals a trade-off in which the textual refusal space provides stronger safety, while the visual refusal space better preserves benign behavior with comparable general multimodal capability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
VLM safety, Jailbreak, low-dimension, utility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2K4NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper argues that effective refusal in VLMs need be neither text-conditioned nor high-dimensional. The authors observe, via singular-value spectra across three VLM families, that visual-harmfulness differences and refusal gradients are both concentrated in low-dimensional subspaces, with refusal more concentrated than harmfulness. Building on this, they construct two fixed rank-16 bases per model: a harmfulness basis Q_rep from SVD of normalized harmful-minus-safe last-token hidden-state differences, and a refusal basis Q_grad from SVD of refusal-margin gradients. A layer is chosen to maximize visual-minus-textual alignment. They then learn only a 16×16 matrix and a bias that maps harmfulness coordinates to refusal coefficients, applied as an input-dependent, rank-constrained hidden-state update at inference. On Qwen2-VL, InternVL3, and LLaVA-1.5, the method reduces MM-SafetyBench ASR by 5.0–18.4 points and HoliSafe unsafe ASR by 10.7–19.2 points, keeps MME nearly unchanged, and largely preserves MOSSBench benign behavior, at much lower compute than the compared defenses. A separate analysis reports a trade-off in which text-conditioned refusal is safer while visual conditioning better preserves benign behavior.

Critical strengths

S1. A cheap, well-specified intervention with a clear efficiency story. Learning a single 16×16 map (272 parameters) over two fixed SVD bases, with a frozen backbone and no auxiliary model or extra VLM pass, is genuinely lightweight, and Table 3 documents the compute advantage over seven baselines fairly.

S2. A clean conceptual separation that is novel in framing. Decomposing control into a harmfulness space (how the input departs from benign) and a refusal space (which directions increase refusal), and learning only the coupling between them, is an interpretable and well-motivated formulation. Driving refusal from a visually conditioned space rather than reusing textual safety representations is a genuinely different design point from most prior VLM defenses.

S3. A useful secondary contribution. The visual-vs-textual refusal-space comparison (Table 4) isolates a safety-versus-benign-behavior trade-off by varying only the conditioning of Q_grad, which is an informative finding in its own right and strengthens the motivation for the visual conditioning choice.

Critical weaknesses

The method is cheap and cleanly designed, and I like the harmfulness-space / refusal-space separation, but the paper is framed as a safety advance that its own tables don’t support (ETA and OH-Defense are safer in Table 2), the absolute ASR after defense is still very high, and the central low-dimensional claim is never tested against the orthogonal-complement attack the paper itself admits it invites. Details as follows:

W1. The safety narrative is not supported by the tables. The paper is framed as a safety defense, but by its own Table 2 it is never the safest method: ETA has lower MM-SafetyBench ASR on all three backbones, OH-Defense is far lower, and on HoliSafe the method is only second-best. The real contribution is a trade-off, not a safety gain.

W2. Absolute safety remains poor. Even after the reported reductions, LLaVA still has 77.8% HoliSafe ASR and Qwen2-VL 50.8% MM-SafetyBench ASR, so most attacks still succeed. The paper leads with percentage-point drops without arguing the practical value at this absolute level.

W3. No adaptive attack, which the method most invites. Appendix C concedes that any perturbation in the orthogonal complement of Q_rep bypasses the rank-16 update, then leaves evaluating this to future work. For a defense whose thesis is that a tiny subspace suffices, this is the key missing experiment.

W4. The low-dimensional claim is under-evidenced. Figure 1 compares only against an isotropic Gaussian, but hidden states are strongly anisotropic with outlier dimensions, so beating that null is weak support. A PCA-spectrum or shuffle control would be the appropriate baseline.

W5. Generalization has a clear counterexample. Qwen3-VL barely changes (MM-SafetyBench 41.5→40.1, HoliSafe 99.8→99.8), showing the single next-token rank-16 space can fail entirely when refusal is less globally shared.

W6. Over-refusal is unstable and model-dependent. MOSSBench refusal rises from 5.3% to 11.7% on Qwen, 3.7% to 19.7% on Qwen2.5-VL, and 10.3% to 22.0% on LLaVA-OneVision, and the method is not trained to control it.

Questions for authors

Please refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2K4N (1/2)

We thank the reviewer for the thoughtful and direct feedback. The comments helped us sharpen the paper’s central message: safety should be evaluated together with utility and inference cost, since a defense that substantially degrades model capability or adds heavy inference overhead has limited practical value. Our method offers a favorable combination of meaningful safety gains, near-vanilla utility, and low additional inference cost (Tables 1–3). We have revised the manuscript to foreground this safety–utility–efficiency trade-off, with the major changes highlighted in blue.

W1. The safety narrative is not supported by the tables.

The central contribution is a strong safety–utility–efficiency trade-off, and we have revised the Introduction and contribution statement to foreground this point. The two methods with clearly lower ASR, OH-Defense and ETA, incur substantial costs in utility and inference efficiency, respectively.

OH-Defense achieves stronger absolute safety (Table 2), but its MME scores drop to 1598.6, 1820.2, and 1378.2, compared with 2317.4, 2385.0, and 1664.0 for ours on Qwen2-VL, InternVL3, and LLaVA-1.5 (Table 1). ETA also achieves lower ASR (Table 2), but requires auxiliary ArmoRM and CLIP models, additional scoring, and conditional Best-of-5 generation (Table 3). Our method therefore provides a favorable operating point with strong safety gains, near-vanilla utility, and minimal inference overhead.

W2. Absolute safety remains poor.

Our goal is to achieve a strong safety–utility–efficiency trade-off, and the reported operating point is chosen with this overall objective in mind. We acknowledge that the remaining ASR is still high in absolute terms, and we have revised the paper to make these absolute values and the corresponding trade-off clearer.

Even when safety is considered on its own, our method outperforms most compared defenses. On Qwen2-VL MM-SafetyBench, most methods remain above 55% ASR, while ours reaches 50.8%; on LLaVA HoliSafe, most remain above 80%, while ours reaches 77.8% (Table 2). The two main methods with substantially lower ASR are OH-Defense and ETA: OH-Defense incurs a large utility drop (Table 1), while ETA requires substantially higher inference cost through auxiliary models and additional generation (Table 3).

If the sole objective is to further reduce ASR, the intervention strength α\alpha provides a direct way to do so. As shown in Figure 4, increasing α\alpha can provide additional safety gains. We use α=1\alpha=1 in the main results because it gives a favorable balance among safety, general utility, and benign behavior.

W3. No adaptive attack against the orthogonal complement.

This is a valid concern under a full white-box threat model, although such an attacker must directly access and manipulate internal hidden states, which is substantially stronger than the input-space attacks available in most practical settings. We therefore evaluated the orthogonal-complement attack at both levels and added the results and discussion to Appendix F.

We selected 50 examples that were originally answered safely under our defense and applied the following adaptive attacks:

Attack spaceAttack settingUnsafe / TotalASR
Hidden-state spaceUnconstrained optimization28 / 5056%
Hidden-state spaceRestricted to Qrep⊥Q_{\mathrm{rep}}^\perp23 / 5046%
Pixel spacePGD, ϵ=8\epsilon=8, no orthogonal penalty14 / 5028%
Pixel spacePGD, ϵ=8\epsilon=8, orthogonal penalty λ=1\lambda=111 / 5022%
Pixel spacePGD, ϵ=8\epsilon=8, orthogonal penalty λ=10\lambda=109 / 5018%

Under direct hidden-state manipulation, the reviewer’s proposed bypass is effective. Restricting the perturbation entirely to Qrep⊥Q_{\mathrm{rep}}^\perp still makes 23/50 previously safe examples unsafe, showing that the low-rank intervention does not provide robustness to an attacker with direct control over internal representations.

The attack becomes substantially less effective when constrained to realizable pixel-space perturbations. At ϵ=8\epsilon=8, standard PGD reaches 28% ASR, while adding the orthogonal-complement objective gives 22% at λ=1\lambda=1 and 18% at λ=10\lambda=10. Across the broader sweep, none of the orthogonally constrained pixel-space attacks outperforms standard PGD.

Overall, the experiment identifies a genuine white-box limitation, while also showing a substantial gap between direct latent manipulation and attacks that must operate through the input image. We now discuss this distinction explicitly in Appendix F and include adaptive robustness as an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 2K4N (2/2)

W4. The low-dimensional claim is under-evidenced.

The reviewer is right that low-dimensional spectral concentration is not specific to harmfulness and refusal. Following the reviewer’s suggestion, we examined the spectra of ordinary model features and gradients and found that they also show strong low-dimensional structure. This makes the original isotropic Gaussian baseline in Figure 1 too weak for arguing that safety-related representations are uniquely low-dimensional, so we removed that baseline and revised the corresponding discussion.

This does not change the motivation of our method. What we need is simply that harmfulness and refusal signals can be represented in compact subspaces, so that control can be carried out efficiently in a low-rank space. We have revised the main text to make this scope explicit and no longer present low-dimensionality as a property unique to harmfulness or refusal.

W5. Qwen3-VL is a counterexample to generalization.

Qwen3-VL is the only clear negative case among the nine VLMs we evaluated, while the other eight models show consistent safety improvements; our new Qwen3.5-9B and text-only LLM results further suggest that this is a model-specific boundary case rather than a general failure to transfer. We have therefore kept the Qwen3-VL result explicit and clarified this limitation in the revised paper.

The additional results support this interpretation across both newer VLMs and text-only LLMs. In the updated Table 7, Qwen3.5-9B improves from 47.2% to 42.1% on MM-SafetyBench and from 52.0% to 46.0% on HoliSafe, while MME remains nearly unchanged (2393.4 → 2384.7) and MOSSBench changes only slightly (6.7% → 7.7%). We also extended the method to four text-only LLMs (Table 9), where HarmBench and SorryBench ASR decrease on all four models and JailbreakBench ASR either decreases or remains unchanged. These broader results make Qwen3-VL the clear exception in our current evaluation rather than the dominant generalization pattern.

W6. Over-refusal is unstable and model-dependent.

Over-refusal is indeed less stable across architectures, but the overall benign refusal rates remain competitive with the compared safety methods. On the three main backbones, our MOSSBench FRR is 11.7%, 5.7%, and 11.3%, substantially lower than AutoSteer (22.7%, 9.3%, and 20.3%) and lower than NullSteer on two of the three models (10.7%, 33.0%, and 51.3%) (Table 2).

The model-dependent increases highlighted by the reviewer are real and deserve explicit discussion. MOSSBench FRR rises from 5.3% to 11.7% on Qwen2-VL, from 3.7% to 19.7% on Qwen2.5-VL, and from 10.3% to 22.0% on LLaVA-OneVision (Tables 1 and 7), showing that benign refusal calibration is not equally stable across architectures.

This level of over-refusal control is encouraging because MOSSBench FRR is not directly optimized by our training objective. The preservation loss encourages the modified model to stay close to the original behavior on benign inputs, but it does not explicitly optimize an over-refusal metric. We have clarified this limitation and identify direct benign-refusal calibration as an important direction for making the trade-off more consistent across models.

Thank you for the detailed response and the revised manuscript. The reframing around the safety–utility–efficiency trade-off, the removal of the isotropic baseline, the additional backbones, and the Qwen3-VL analysis make the claims better aligned with the evidence, and I have raised my soundness score accordingly. However, the main concerns were addressed by narrowing the claims rather than by new evidence that overturns them: the method’s absolute safety remains mid-range among the compared defenses, the low-dimensional observation is no longer specific to harmfulness or refusal, and the adaptive-attack results in Appendix F lack an undefended reference, so they do not yet establish robustness. I therefore keep my overall rating.

Dear Reviewer 2K4N,

Thank you again for your careful and insightful review. In the rebuttal, we directly addressed the key issues you identified: most importantly, we evaluated the orthogonal-complement attack in both hidden-state and realizable pixel space, added stronger controls for the low-dimensionality analysis, revised the framing of the safety–utility trade-off, and added results on newer VLM backbones.

We believe these additions materially strengthen the support for our core claims and make the scope and positioning of the paper more precise. We would greatly appreciate it if you could revisit our response and revised manuscript, and reconsider your overall assessment and rating based on the revised framing and strengthened support for the paper’s central claims.

Thank you very much for your time.

Reviewer_KiB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new method to detect harmful visual input and steer outputs to be safe. The proposed approach is to train a linear layer to act as harmful-to-refusal mapping.

Critical strengths
  • The proposed method is very lightweight which only requires 272 parameters.
  • The amount of training data required is small, which includes 799 pairs.
Critical weaknesses
  • The proposed method is similar to the steering vector but with an additional harmful-to-refusal mapping. The motivation for using such a mapping is not well justified. Compared to steering vector-based methods, is the main advantage of the proposed method reducing the amount of training data?
  • One of the main claimed benefits of the proposed method is the light training, which requires only 799 pairs. But it is also known that the generalization of steering vector-based methods depends on the quality of training data. How well does the proposed method generalize?
  • The empirical improvement in terms of safety appears not significant compared to existing methods.
  • The models used in the paper (Qwen2-VL, LLaVA-1.5, InternVL3) are relatively old. Appendix B includes some results on newer models, but those are only evaluated on a subset of benchmarks. It will make the experiment much stronger by using newer models for the main evaluation.
Questions for authors
  • Does the observation that harmfulness and refusal are in a low-dimensional subspace apply to LLMs too? The proposed harmful-to-refusal mapping seems applicable to LLMs as well.
3 comments
AuthorsOpenReview ↗

Response to Reviewer KiB5 (1/2)

We thank the reviewer for the helpful comments. The feedback prompted us to add direct steering and data-size ablations, extend the evaluation to Qwen3.5 and text-only LLMs, and clarify where the main advantage of our method lies. The major revisions are highlighted in blue in the updated manuscript.

1. Motivation for the harmfulness-to-refusal mapping (W1)

The main advantage of our mapping is not simply reducing the amount of training data; it is achieving stronger safety while preserving general utility through targeted, input-dependent control. Harmfulness and refusal are represented in separate compact spaces, and the learned mapping converts the harmfulness state of each input into an update restricted to refusal-sensitive directions. This gives the intervention enough flexibility to respond to the current input without broadly steering the hidden representation.

In the main evaluation, this design substantially reduces ASR while keeping MME close to the original models (Table 1). We also compared it directly with traditional harmfulness steering on the same reduced evaluation panel:

ModelMethodMM-S ASR ↓MME ↑
Qwen2-VLTraditional steering65.872344.5
Ours50.502528.3
InternVL3Traditional steering58.882370.3
Ours44.312377.8
LLaVA-1.5Traditional steering64.071736.5
Ours47.501685.1

The learned mapping is consistently much more effective than conventional steering, reducing MM-SafetyBench ASR by 14.57–16.57 percentage points while retaining competitive utility across the three models. This directly supports the motivation for mapping harmfulness into a dedicated refusal space instead of applying a generic steering direction. The 272 trainable parameters and modest training-data requirement are additional efficiency benefits of this design.

2. Generalization across training-data sizes (W2)

To test how strongly the method depends on the amount of training data, we retrained it with 200, 400, and the full 799 training pairs while keeping the model, training procedure, and evaluation panel fixed. The method remains effective even with only 200 pairs, and the variation with data size is moderate and non-monotonic.

ModelMetric200 pairs400 pairs799 pairs
Qwen2-VLMM-S ASR ↓49.1053.0950.50
MME ↑2453.32453.32528.3
InternVL3MM-S ASR ↓47.7046.7144.31
MME ↑2370.32370.32377.8
LLaVA-1.5MM-S ASR ↓47.9054.2947.50
MME ↑1710.11731.51685.1

The key observation is that strong performance does not require the full 799-pair training set. Even with 200 pairs, the method already achieves substantial safety gains across all three models. The 799 pairs are simply the predefined training split, we did not select examples based on downstream performance. This also leaves room for further improvement through more carefully curated or higher-quality training pairs.

AuthorsOpenReview ↗

Response to Reviewer KiB5 (2/2)

3. The safety improvement is not significant compared with existing methods (W3)

If safety is considered in isolation, our method is not always the strongest, and we have revised the manuscript to state this clearly. The main strength is the overall safety–utility–efficiency trade-off. Among the compared methods, the two prominent approaches with clearly stronger absolute safety are OH-Defense and ETA, and both achieve that gain with substantial additional cost.

OH-Defense obtains lower ASR at the cost of a large drop in general utility. Its MME scores are 1598.6, 1820.2, and 1378.2, compared with 2317.4, 2385.0, and 1664.0 for ours on Qwen2-VL, InternVL3, and LLaVA-1.5 (Table 1).

ETA provides a more modest safety advantage but requires considerably heavier inference. Its MM-SafetyBench ASR is 5.0, 3.8, and 3.5 percentage points lower than ours on the three models, respectively, while requiring auxiliary ArmoRM and CLIP models, additional safety scoring, and conditional Best-of-5 generation (Tables 2–3).

These costs are important when judging the practical value of a safety method. Our approach provides meaningful safety improvements while keeping general capability close to the original model and requiring no auxiliary model, additional full VLM pass, or extra candidate generations at inference. We have revised the paper to make this trade-off, rather than absolute safety alone, the central framing of our contribution.

4. Evaluation on newer VLMs (W4)

We added Qwen3.5-9B to Table 7, and the results show that the method continues to provide a favorable safety–utility trade-off on this newer model generation. We report all four evaluation metrics:

MetricVanillaOurs
MM-SafetyBench ASR ↓47.2%42.1%
HoliSafe ASR ↓52.0%46.0%
MME ↑2393.42384.7
MOSSBench FRR ↓6.7%7.7%

On Qwen3.5-9B, our method reduces ASR on both safety benchmarks with only small changes in utility and benign refusal. MM-SafetyBench ASR decreases by 5.1 points and HoliSafe by 6.0 points, while MME changes by only 8.7 points and MOSSBench FRR by 1.0 point.

Together with the existing results on Qwen2.5-VL, LLaVA-1.6, and LLaVA-OneVision, this extends the evidence beyond the three original backbones and across newer model generations and scales. We also report the weaker Qwen3-VL result explicitly, as discussed in the manuscript, since the effectiveness remains architecture-dependent.

5. Applicability to text-only LLMs (Q1)

The same harmfulness-to-refusal mapping also transfers to text-only LLMs, providing consistent safety improvements across four different model families. We added these results to Appendix E and evaluated Llama-3.1-8B, Qwen3.5-9B, Ministral-3-8B, and Granite-4.2-8B on HarmBench, JailbreakBench, and SorryBench, together with JailbreakBench FRR.

ModelHarmBench ASR ↓JailbreakBench ASR ↓SorryBench ASR ↓JailbreakBench FRR ↓
Llama-3.1-8B30.00 → 26.508.00 → 6.0025.91 → 23.4111.00 → 11.00
Qwen3.5-9B5.00 → 3.750.00 → 0.0015.00 → 11.3630.00 → 30.00
Ministral-3-8B24.07 → 17.055.00 → 1.0043.41 → 38.8613.00 → 12.00
Granite-4.2-8B3.13 → 1.251.00 → 0.0015.23 → 12.9522.00 → 24.00

The safety trend is consistent across the four LLMs: HarmBench and SorryBench ASR decrease on every model, while JailbreakBench ASR either decreases or remains unchanged. The changes in over-refusal are generally modest, including unchanged FRR on Llama-3.1 and Qwen3.5 and a slight reduction on Ministral, although Granite shows a small increase.

These results provide direct behavioral evidence that the mapping itself is not specific to multimodal architectures. A dedicated spectral analysis on text-only LLMs would be needed to determine whether their harmfulness and refusal representations exhibit the same low-dimensional structure observed in the VLM experiments.

Dear Reviewer KiB5,

Thank you again for your thoughtful feedback. In the rebuttal, we directly validated the necessity of the harmfulness-to-refusal mapping, and further strengthened the evidence for generalization through data-scaling experiments and evaluations on newer VLM backbones.

We hope these results fully address the main concerns raised in your review, and we would greatly appreciate it if you could revisit our response and reconsider your overall assessment and rating.

Thank you very much for your time.

Reviewer_cJ3gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that effective refusal need be neither text-conditioned nor high-dimensional, and visual harmfulness and refusal can both be captured in low-dimensional subspaces. It propose a visually conditioned harmfulness-to-refusal mapping that connects the two low-dimensional spaces with only 272 trainable parameters.

Critical strengths
  1. The method is lightweight with only 272 trainable parameters.
  2. Separating harmfulness and refusal into two low-dimensional subspaces is intuitive, and the spectral analysis (Figure 1) is an interesting empirical observation.
  3. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Which layer ℓ* is selected for each model? The paper provide no empirical guidance for this choice. Also, do the spectral concentration of Q_grad and the optimal rank change with the size of the anchor set?
  2. The authors claim refusal “need not be text-conditioned,” but Table 4 shows the textual refusal space is consistently stronger in safety.
  3. The method performs poorly on newer models such as Qwen3-VL, as the authors acknowledge, which raises doubts about its generality.
  4. Evaluation covers only static benchmarks e.g., MM-SafetyBench and HoliSafe. More attack types should be evaluated, e.g., FigStep [3].
  5. Similar works [1, 2] are not cited or compared, e.g., VLM-Guard [1], which uses a low-dimensional subspace and a safety steering direction.
  6. (Minor) Values are inconsistent. The vanilla Qwen2-VL overall MM-SafetyBench ASR is 63.4 in Table 2 but 62.6 in Tables 5 and 6.
  7. (Minor) The authors only use seed 0, with no multiple trials, as stated in the implementation details.

[1] VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap [2] Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models [3] FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts

Questions for authors
  1. How were α and λ_pres chosen? Were they tuned on the test benchmarks?
  2. The compliance anchors include “Yes” and the refusal anchors include “No”, while MME consists entirely of Yes/No questions. The validity of MME as a utility metric is therefore questionable.
5 comments
AuthorsOpenReview ↗

Response to Reviewer cJ3g (1/3)

We thank the reviewer for the constructive comments. The feedback helped us clarify several methodological choices and strengthen the evaluation, particularly on layer and anchor sensitivity, newer models, additional attacks, and reproducibility. The major revisions are highlighted in blue in the updated manuscript.

1. Layer selection and anchor-set size (W1)

We have clarified how the intervention layer is selected and now report the selected layer for every backbone in Appendix A (Table 5). For the three primary models, the selected layers are 13 for Qwen2-VL, 19 for InternVL3, and 19 for LLaVA-1.5. For each candidate layer, we compute visual and text-conditioned alignment scores on 100 held-out validation pairs and select the layer maximizing their difference; no test benchmark is used for this choice.

The refusal-gradient space remains strongly low-dimensional as the anchor set becomes smaller, although its spectrum becomes moderately less concentrated. We tested Qwen2-VL with the original anchor sets (14 refusal / 13 compliance phrases), half-sized sets (7/7), and small sets (3/3), while keeping the training data, intervention layer, and evaluation panel fixed.

Anchor setRank-4 massRank-8 massRank-16 massRank-32 massRank for 90% mass
Full (14/13)77.28%82.26%86.61%90.18%31
Half (7/7)74.62%80.33%85.31%89.28%37
Small (3/3)72.71%78.77%83.88%88.20%45

The downstream results show a similar picture: performance is not highly sensitive to using a large refusal subspace, and a single rank should not be viewed as universally optimal. On the same 501-example MM-SafetyBench panel, rank 4 gives the lowest ASR for all three anchor sizes, while higher ranks can provide better utility depending on the setting.

Anchor setRankMM-S ASR ↓MOSS FRR ↓MME ↑
Full (14/13)449.10%7.78%2349.9
854.49%7.78%2444.5
1649.30%12.22%2453.3
3249.50%11.11%2378.3
Half (7/7)448.50%7.78%2428.3
853.69%8.89%2449.9
1651.50%10.00%2441.1
3250.90%10.00%2453.3
Small (3/3)447.90%10.00%2449.9
853.49%12.22%2369.5
1652.89%11.11%2444.5
3252.10%8.89%2445.8

These results suggest that anchor-set size changes the exact spectrum and safety–utility operating point, but does not remove the low-dimensional structure that our method relies on. Rank 16 remains a compact and effective setting in the main experiments, while the sensitivity study shows that smaller ranks can also work well and that the best rank depends on which aspect of the safety–utility trade-off is prioritized.

2. The authors claim refusal “need not be text-conditioned,” but Table 4 shows the textual refusal space is consistently stronger in safety. (W2)

Our claim that refusal “need not be text-conditioned” is about feasibility: visual conditioning alone can drive effective refusal, even though text conditioning can provide stronger absolute safety. Table 4 makes this trade-off clear.

ModelVisual ASR ↓Textual ASR ↓Visual MOSS ↓Textual MOSS ↓
Qwen2-VL50.845.411.717.0
InternVL346.644.35.78.0
LLaVA-1.547.647.311.311.7

Text conditioning is safer on MM-SafetyBench, while visual conditioning consistently reduces benign over-refusal and retains comparable MME utility. We therefore use the visual refusal space as the main setting because it provides a favorable preservation-oriented safety–utility trade-off. We have revised the wording to make clear that “need not be text-conditioned” does not imply that visual conditioning always achieves the lowest ASR.

AuthorsOpenReview ↗

Response to Reviewer cJ3g (2/3)

3. Generalization to newer VLMs (W3)

The expanded evaluation shows positive transfer across several newer VLM generations, with Qwen3-VL remaining the one clear exception. We added newer Qwen and LLaVA-family models to Table 7, including Qwen3.5-9B and a 32B backbone.

ModelVanilla MM-S ASR ↓Ours MM-S ASR ↓
Qwen2.5-VL-7B53.232.6
Qwen2.5-VL-32B57.943.1
Qwen3.5-9B47.242.1
LLaVA-1.6-7B67.250.9
LLaVA-OneVision-Qwen2-7B56.037.8

These results show that the method transfers beyond the three original backbones and remains effective across newer model generations and scales. This includes Qwen2.5-VL-32B and Qwen3.5-9B.

Qwen3-VL-8B remains an important boundary case, and we report it explicitly. Its MM-SafetyBench ASR changes only from 41.5% to 40.1%, with essentially no improvement on HoliSafe. We discuss this limitation in the revised manuscript and avoid claiming universal effectiveness. The broader results nevertheless show that this behavior is the exception in our current cross-model evaluation.

4. Additional attack evaluation: FigStep (W4)

We added a FigStep evaluation as suggested. We use the FigStep subset of JailBreakV-28K with 2,000 inputs per model and evaluate responses with LlamaGuard-7B under the benchmark's 16-category safety policy.

ModelVanilla ASR ↓Ours ASR ↓
Qwen2.5-VL2.40%0.95%
Qwen3-VL0.60%0.45%
InternVL34.30%4.40%

The method improves FigStep robustness on Qwen2.5-VL and slightly on Qwen3-VL, while InternVL3 changes only marginally in the opposite direction. The vanilla ASRs are already very low on this evaluation, leaving limited room for large absolute improvements.

5. Related low-dimensional steering methods (W5)

We added both methods suggested by the reviewer to the related-work discussion. Since neither VLM-Guard nor Principled Steering via Null-space Projection provides a public implementation, we reimplemented the latter method (NullSteer) ourselves and evaluated it under the same setup as our method. The results are now included in Tables 1–2.

Under this matched evaluation, our method provides a stronger overall safety–utility–over-refusal trade-off than NullSteer. On HoliSafe, ASR decreases from 75.7% to 72.2% on Qwen2-VL, 78.8% to 75.6% on InternVL3, and 93.0% to 77.8% on LLaVA-1.5, while our method also achieves higher MME scores on all three backbones. The difference in benign behavior is especially large on InternVL3 (5.7% vs. 33.0% MOSSBench refusal) and LLaVA-1.5 (11.3% vs. 51.3%).

NullSteer is slightly stronger on MM-SafetyBench for Qwen2-VL and InternVL3, but this comes with a less favorable overall trade-off on the other metrics. Our method uses an input-dependent mapping from visual harmfulness coordinates to refusal coefficients, allowing the intervention to adapt to each input while preserving benign behavior and general utility more effectively.

AuthorsOpenReview ↗

Response to Reviewer cJ3g (3/3)

6. Benchmark subsets and multiple random seeds (W6, W7)

The apparent discrepancy in Qwen2-VL vanilla ASR comes from evaluating different benchmark sizes, and we have made this explicit in the revised manuscript. The 63.4% value in Table 2 is measured on the full MM-SafetyBench benchmark, while 62.6% in Table 7 comes from the sampled subset used for the broader-model evaluation.

The safety results are also stable across training seeds. We repeated training with four seeds (0–3) and evaluated them on the same 501-example MM-SafetyBench panel.

ModelSeed-0 ASR ↓Four-seed ASR mean ± SD ↓Difference from seed 0Four-seed MME mean ± SD ↑
Qwen2-VL50.50%50.40 ± 0.20%−0.10 pp2482.32 ± 49.18
InternVL344.31%44.71 ± 1.02%+0.40 pp2374.08 ± 4.33
LLaVA-1.547.50%47.26 ± 1.49%−0.25 pp1706.48 ± 33.86

The mean ASR changes by at most 0.40 percentage points relative to seed 0, indicating that the reported safety gains are not driven by a favorable initialization. The standard deviations reflect variation across training seeds on the same fixed evaluation panel.

7. Hyperparameter selection and MME Yes/No validity (Q1, Q2)

The main hyperparameters were selected without using the test benchmarks. We use α=1\alpha=1 and λpres=5\lambda_{\mathrm{pres}}=5 as described in Section 4.1. Appendix B evaluates α∈{0,0.5,1,2}\alpha \in \{0, 0.5, 1, 2\}, showing the expected safety–over-refusal trade-off as intervention strength increases. Appendix C evaluates λpres∈{0,2.5,5,7.5,10}\lambda_{\mathrm{pres}} \in \{0, 2.5, 5, 7.5, 10\}; at λpres=5\lambda_{\mathrm{pres}}=5, refusal-token rates on unsafe and safe VSCBench examples are 48% and 9%, respectively.

The inclusion of “Yes” and “No” in the anchor sets does not produce a meaningful systematic bias on MME. These anchors are used to define the training surrogate, while MME is evaluated from the model's generated answers. To verify this directly, we performed a paired audit on 2,374 MME questions per model:

ModelVanilla accuracyOurs accuracyChanged answers“Yes” predictions, Vanilla → Ours
Qwen2-VL88.332%88.416%10/23741208 → 1204
InternVL388.500%88.543%5/23741108 → 1109
LLaVA-1.576.495%76.327%20/23741455 → 1469

Only 5–20 answers change out of 2,374 per model, with accuracy shifting by at most 0.17 percentage points and essentially unchanged Yes/No prediction counts. This provides direct evidence that the intervention does not introduce a large systematic Yes/No response bias on MME.

We appreciate the reviewer’s suggestions, they helped us clarify the method, broaden the evaluation, and make the scope and limitations of the results more explicit.

Reviewer_CsQiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that visual harmfulness and refusal are concentrated in low-dimensional subspaces, with refusal exhibiting stronger spectral concentration. It proposes a visually conditioned harmfulness-to-refusal mapping with only 272 trainable parameters, keeping the VLM backbone frozen. Experiments on three VLMs show reduced attack success on MM-SafetyBench and HoliSafe with nearly unchanged MME performance. Comparing visual and textual refusal spaces reveals a trade-off: textual control provides stronger safety, while visual control better preserves benign behavior.

Critical strengths
  1. The method is well motivated by the observed low-dimensional structure of harmfulness and refusal. Learning a mapping between these spaces provides a simple way to translate visual harmfulness signals into refusal, using only 272 trainable parameters while keeping the backbone frozen.

  2. The evaluation considers safety, over-refusal, and general multimodal ability together. Across the three main backbones, attack success rates decrease while MME performance remains nearly unchanged. The comparison of visual and textual refusal spaces also clarifies the trade-off between stronger safety and preserving benign behavior.

Critical weaknesses
  1. AutoSteer is cited but not evaluated, despite also using internal risk signals to trigger refusal with a frozen backbone. This leaves the advantage over closely related conditional steering unclear.

  2. The ablations do not establish the need for the proposed subspaces and input-dependent mapping. Comparisons with an input-independent refusal shift and random subspaces of the same rank are missing.

  3. The organization and writing need improvement. Section 3.2 does not explain how the alignment scores for layer selection are computed, and the paper lacks a conclusion summarizing its findings, limitations, and scope.

  4. The submission provides no code link or implementation files. Together with missing methodological details, this makes the main results difficult to reproduce independently.

Questions for authors
  1. How does your method compare with AutoSteer in safety, over-refusal, and general utility under the same evaluation setup?

  2. Have you compared the proposed mapping with an input-independent refusal shift and random subspaces of the same rank, using the same training data and objectives?

  3. How are the visual and textual alignment scores in Eq. (3) computed, and why is their difference used to select the intervention layer?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CsQi (1/2)

We thank the reviewer for the constructive feedback. We have strengthened the experimental comparisons, clarified the methodology, and improved the presentation accordingly. The major changes are highlighted in blue in the revised manuscript.

1. Comparison with AutoSteer (W1, Q1)

Under the same evaluation setup, our method provides a better overall safety–utility trade-off than AutoSteer. We have added AutoSteer to Tables 1–2 and evaluated it across all three backbones.

ModelMethodMM-S ASR ↓HoliSafe ASR ↓MOSS refusal ↓MME ↑
Qwen2-VLAutoSteer59.450.722.71994.2
Ours50.872.211.72317.4
InternVL3AutoSteer48.476.79.32373.5
Ours46.675.65.72385.0
LLaVA-1.5AutoSteer54.976.320.31654.8
Ours47.677.811.31664.0

Across MM-SafetyBench, MOSSBench, and MME, our method performs better on all three backbones. AutoSteer is stronger on HoliSafe for Qwen2-VL and slightly stronger for LLaVA-1.5, showing that the safety advantage is benchmark-dependent.

The main difference is that our method uses the input-specific visual harmfulness representation itself to determine the refusal intervention. It maps harmfulness coordinates directly into a low-dimensional refusal space with only 272 trainable parameters, giving a favorable balance among safety, benign behavior, and general utility.

2. Necessity of the subspaces and input-dependent mapping (W2, Q2)

The additional ablations show that the learned input-dependent mapping consistently provides substantially stronger safety than simpler steering alternatives. We compare against three controls:

  • Traditional harmfulness steering: conventional harmfulness-based activation steering without the learned mapping.
  • B-only (input-independent): removes the input-dependent term and leaves a fixed refusal update, Δh=Qgradb\Delta h = Q_{\mathrm{grad}}b.
  • Random refusal subspace: replaces the learned refusal basis with a rank-matched random basis while retaining the input-dependent mapping.
ModelMethodMM-S ASR ↓MOSS refusal ↓MME ↑
Qwen2-VLTraditional steering65.875.562344.5
B-only57.298.892453.3
Random refusal subspace59.487.782528.3
Ours50.5010.002528.3
InternVL3Traditional steering58.884.442370.3
B-only49.506.672370.3
Random refusal subspace47.707.782362.8
Ours44.317.782377.8
LLaVA-1.5Traditional steering64.0711.111736.5
B-only61.0810.001702.6
Random refusal subspace62.0810.001608.7
Ours47.508.891685.1

The main result is that the full mapping is clearly more effective at reducing ASR than all three simpler alternatives across the three backbones. Relative to the input-independent B-only control, it reduces ASR by 6.79, 5.19, and 13.58 points; replacing the learned refusal basis with a random rank-matched subspace also consistently weakens safety (+8.98, +3.39, and +14.58 ASR points). These results support both components of the design: input-dependent control and learned refusal-sensitive directions.

The trade-off in over-refusal is more nuanced. Fixed steering can sometimes yield lower MOSSBench refusal, but this comes with substantially weaker safety, while our mapping achieves the strongest ASR reduction overall.

For completeness, the B-only result is a diagnostic obtained by zeroing the input-dependent matrix in a trained random-representation checkpoint; it is not an independently retrained fixed-shift baseline.

3. Layer selection and presentation (W3, Q3)

We have revised Section 3.2 to make the layer-selection criterion explicit and expanded Appendix A to report the selected layer for each model. For each candidate layer, we compute visual and textual alignment scores over 100 held-out validation pairs. Each score is the average normalized inner product between the harmful–safe representation displacement projected onto the learned representation subspace and the refusal-margin gradient.

We select the layer where refusal sensitivity is most strongly associated with visual harmfulness relative to text-conditioned safety signals. This is captured by the difference between the visual and textual alignment scores in Eq. (3).

We also improved the manuscript organization and added Section 6 (Conclusion) to summarize the main findings, limitations, and scope.

AuthorsOpenReview ↗

Response to Reviewer CsQi (2/2)

4. Code availability and reproducibility (W4)

We have provided an anonymized code repository and expanded the implementation details to make the method easier to reproduce.

https://anonymous.4open.science/r/vlm_mapping_review-6D15/

The revised manuscript now specifies the data split, subspace construction, layer selection, training configuration, and inference procedure in greater detail.

Dear Reviewer CsQi,

Thank you again for your constructive feedback. We have addressed the main concerns in your review, including the AutoSteer comparison, the input-independent and random-subspace ablations, and the missing methodological and reproducibility details.

Since these points formed the main basis of your concerns, we would greatly appreciate it if you could take another look at our response and revised manuscript. If you find that these additions adequately address your concerns, we would also be grateful if you could reconsider your assessment and rating.

Thank you very much for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.