acceptodds
Under review as a conference paper at ICLR 2027

Aligning Refusal Behavior with Safety Awareness in Vision-Language Models

Abstract

Vision Language Models (VLMs) have shown remarkable cross-modal reasoning capabilities, yet their safety alignment remains a critical challenge. In this work, we address a fundamental gap in existing VLMs: while these models internally represent harmful content as distinguishable from benign inputs, this safety awareness does not reliably translate into output behavior. Unlike prior approaches that detect and block harmful inputs externally via auxiliary models or additional forward passes, we propose REBASE, a training-free inference-time method that, within a single forward pass, directly bridges the underlying misalignment between the model's internal safety representations and its refusal or compliance behavior through activation steering. Experiments across diverse model architectures and benchmarks demonstrate that our method significantly reduces attack success rates across all settings while maintaining or even improving model utility on benign samples and general vision-language tasks, with less than 2% additional latency. An anonymized version of the code is provided in the supplementary material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
VLMs, Safety alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WqRcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • REBASE addresses the gap between recognizing harmful multimodal inputs and producing appropriate refusals. It extracts harmful/benign activation prototypes and a refusal/compliance steering vector, then uses a standardized projection and asymmetric tanh coefficient to adjust generation without updating model weights. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B report improved attack success rate (ASR), benign non-refusal and MM-Vet performance, with cross-dataset transfer and steering-magnitude ablations. The central claims are that directional misalignment explains safety failures and that correcting it requires negligible inference overhead.
Critical strengths
  • Table 1 reports a meaningful improvement in the joint safety/utility trade-off. On LLaVA/VLGuard, ASR decreases from 22.85% to 2.04%, while benign non-refusal remains 99.28%; on Falcon, REBASE improves this latter metric from AutoSteer's 75.96% to 89.93% at similar ASR. Automating Steering for Safe Multimodal Large Language Models trains a safety prober and refusal head, whereas REBASE uses activation statistics. Obtaining comparable protection with this simpler controller would be valuable if the matched evaluation is confirmed.
  • I find the transfer and magnitude experiments useful. Table 3 transfers vectors in both directions between VLGuard and Falcon, with benign-score losses no larger than 2.63 points, while §4.6.1 tests continuous magnitude against a sign-based controller. Table 2 additionally checks answer correctness on MM-Vet instead of relying entirely on refusal rates. Together these experiments provides evidence that the method can preserve useful behavior, even though they do not yet isolate every proposed mechanism.
Critical weaknesses
  • Eq. 3 changes input harmfulness and response behavior together, so the extracted direction and PCA plots do not establish causal misalignment. Programming Refusal with Conditional Activation Steering already conditions refusal and SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models adapts its strength; their text-only and classifier-based designs differ, but missing matched comparisons leave REBASE’s specific contribution unclear.
  • Appendix B includes safe-image/unsafe-instruction cases, but the released VLGuard loader excludes them; Falcon evaluation also filters by reference-response content (prompting_with_steering.py:113; generate_vectors.py:198–208,293–347). These selection rules leave Table 1’s coverage and baseline comparability uncertain until the actual evaluated IDs and subgroup counts are provided.
  • I could not reconcile §4.5’s single-pass claim with prompting_with_steering.py:220–239,431–453: it probes, resets, then generates without passing a cache. The timer excludes the probe, so the released implementation does not substantiate Figure 8’s 0.01–0.02-second overhead; end-to-end measurements from the actual experiment are needed.
  • Appendix A counts gibberish as refusal and substantive non-refusal as helpful, while the safety judge never sees the image. MM-Vet only partly addresses this; unlike the separate outcomes in Refusal in Language Models Is Mediated by a Single Direction, the reported metrics cannot distinguish coherent refusal, degeneration, and harmful compliance.
  • Table 3 tests dataset transfer, but defense-aware attacks, as evaluated in Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks, are absent. Falcon ASR gains over AutoSteer are only 0.08/0.04 percentage points without paired intervals, leaving adaptive robustness and ranking uncertain despite larger benign-utility gains.
Questions for authors
  • Required clarification: Which exact sample IDs and retained subgroup counts produced Table 1, and were the VLGuard and Falcon filters above applied identically to every baseline?
  • Required clarification and validation: Which code produced Figure 8, where is the probe cache reused, and what overhead remains when timing the complete defended request with output lengths controlled?
  • Required evidence: Do matched-input response contrasts and full-dimensional interventions support the misalignment explanation, and does REBASE retain an advantage over matched-budget CAST and SafeSteer?
  • Required validation: How often are reported “refusals” incoherent or non-actionable harmful responses, and how do image-aware human judgments change ASR, benign correctness and method rankings?
  • Required validation: What are the paired confidence intervals and defense-aware attack results, including stablity across extraction samples? These would materially affect confidence in the claimed robustness rather than simply extend the application scope.
2 comments

Dear Reviewer WqRc,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

AuthorsOpenReview ↗

Warm Reminder - One Day Left in the Discussion Period

Dear Reviewer WqRc,

As the discussion phase ends in less than one day, we kindly ask that you let us know whether we have addressed your concerns, and if so, we would appreciate it if you would consider raising the score. Thank you for your time and effort.

Best regards,

Authors

Reviewer_wJW7OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the gap between safety-relevant representations and refusal behavior in VLMs. It proposes an inference-time activation-steering method that requires no model parameter updates. The method constructs harmful and benign centroids using activations at the last image token and the assistant token, and estimates input harmfulness by projecting onto the direction that separates these centroids. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B report improvements on VLGuard and Falcon, generally preserved MM-Vet performance, and low inference overhead.

Critical strengths
  1. The proposed method is simple. REBASE requires only a projection, scalar transformation, and activation addition during inference. It reuses activations from the normal prefill pass and avoids a separately trained auxiliary network.

  2. On VLGuard, REBASE reduces LLaVA’s ASR from 22.85% to 2.04%, while benign accuracy remains approximately unchanged. On Falcon, it achieves similar ASR to AutoSteer -- 9.58% v.s. 9.66% -- with substantially higher benign accuracy, 89.93% v.s. 75.96%.

  3. The paper includes useful analyses beyond the main benchmark comparison. Cross-dataset transfer, evaluation across five jailbreak categories, and ablations of steering magnitude and query conditioning help make the conclusions convincing.

Critical weaknesses
  1. The main concern is the repeated claim that safety failures stem from directional misalignment between safety awareness and refusal behavior.. The evaluation on PCA separation and successful steering cannot establish that directional misalignment causes safety failures.

  2. There still lacks direct comparisons with other adaptive steering methods, like SafeSteer.

Questions for authors

Please refer to the weaknesses.

2 comments

Dear Reviewer wJW7,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

AuthorsOpenReview ↗

Warm Reminder - One Day Left in the Discussion Period

Dear Reviewer wJW7,

As the discussion phase ends in less than one day, we kindly ask that you let us know whether we have addressed your concerns, and if so, we would appreciate it if you would consider raising the score. Thank you for your time and effort.

Best regards,

Authors

Reviewer_yKE2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the gap between safety-relevant internal representations and refusal behavior in vision-language models. It proposes REBASE, an inference-time intervention that estimates input harmfulness from contrastive activation statistics and dynamically scales a refusal steering vector during generation without updating model parameters. Experiments on two VLMs report reduced attack success rates on VLGuard and Falcon, generally preserved performance on MM-Vet, and low additional inference latency.

Critical strengths
  1. REBASE achieves the lowest ASR and highest SCS among the evaluated methods on both models in Table 1.

  2. The method requires no backbone updates or auxiliary neural network and adds only 0.01–0.02 seconds of inference latency per sample.

  3. The MM-Vet results, cross-dataset transfer experiments, and five jailbreak categories provide additional evidence of utility preservation and generalization.

Critical weaknesses
  1. REBASE lacks evaluation against adaptive attacks targeting the defended model. Under Eq. (11), harmful inputs assigned negative risk scores receive steering away from refusal, potentially encouraging compliance. The existing attack-category evaluation does not establish robustness to this failure mode.

  2. The evaluation conflates refusal with irrelevant or degenerate outputs, while BA measures non-refusal rather than answer correctness.

  3. The paper does not sufficiently ablate several key design choices, including unconditional steering, zero intervention for negative risk scores, and individual token sources for key vector construction. Furthermore, comparisons with closely related recent methods, such as SafeSteer and JRS-Rem, are missing.

  4. Although a 10% held-out validation split is described, the paper does not clearly specify whether samples used to construct steering and key vectors are disjoint from the validation split.

Questions for authors
  1. How are the axes in Figures 2 and 4 constructed? With prompt activations unchanged, does Figure 4 reflect representation changes or changes in response labels?

  2. Does the safety evaluator receive image information? Please separately report refusals, harmful responses, and degenerate outputs, and assess answer correctness on benign safety-benchmark samples.

  3. How does REBASE perform under adaptive attacks, especially when harmful inputs receive negative risk scores? Please compare against unconditional steering, zero intervention for negative scores, and image-only or ASSISTANT-token-only keys.

  4. What samples and responses are used to construct the vectors, and are validation samples excluded? Are transfer hyperparameters selected only on the source domain? Please provide uncertainty estimates for the main results.

  5. What explains the MM-Vet utility gains, particularly LLaVA’s Knowledge increase from 0.4119 to 0.6071? Does disabling reverse steering remove these gains, and do they reflect fewer refusals, better answers, or evaluation variability?

2 comments

Dear Reviewer yKE2,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

AuthorsOpenReview ↗

Warm Reminder - One Day Left in the Discussion Period

Dear Reviewer yKE2,

As the discussion phase ends in less than one day, we kindly ask that you let us know whether we have addressed your concerns, and if so, we would appreciate it if you would consider raising the score. Thank you for your time and effort.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.