acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Recovering Safety Neurons in VLMs Against Multimodal Jailbreak

Abstract

Vision-language models (VLMs) inherit safety alignment from their language backbones, yet visual inputs can undermine this alignment and enable harmful requests to bypass refusal behavior. We uncover a previously uncharacterized neuron-level phenomenon, termed vision-induced safety-neuron suppression, in which visual inputs may suppress neurons associated with harmful-request refusal and shift their activation patterns toward those induced by benign inputs. Motivated by this finding, we introduce VISAR (Visual Intervention for Safety-neuron Activation Restoration), a lightweight visual intervention framework that uses a differentiable safety probe to supervise a plug-and-play SuffixViT for generating corrective visual suffixes, restoring the suppressed refusal-aligned safety state without modifying the pretrained VLM. Experiments across multiple VLMs and three multimodal jailbreak attacks, compared with four representative defenses, show that VISAR substantially reduces harmful responses while preserving benign utility. Mechanistic analyses further confirm that improved safety is accompanied by restoration of the refusal-aligned safety-neuron activation pattern, demonstrating that visual-side restoration of suppressed safety states provides an effective approach to mitigating multimodal safety degradation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Safety Alignment, Model Interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CpF9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies vision-induced safety-neuron suppression in VLMs, where visual inputs weaken refusal-related neuron activations and increase vulnerability to multimodal jailbreaks. This paper proposes VISAR, which uses a learned visual suffix guided by a safety-neuron probe to restore refusal-aligned activations without modifying the pretrained VLM.

Critical strengths
  1. This work provides a neuron-level perspective on multimodal safety degradation, revealing how visual inputs affect refusal-related activations.
  2. VISAR restores safety-related activations through visual suffix tokens without modifying pretrained VLM parameters, introducing minimal inference overhead.
  3. Experiments across three VLMs and multiple jailbreak benchmarks demonstrate substantial reductions in attack success rates while largely preserving benign performance.
Critical weaknesses
  1. This work only establishes a correlation between safety-neuron activations and refusal behavior, without direct neuron-level interventions to validate their causal role.
  2. The experiments focus on relatively outdated VLMs, leaving the generalizability of the proposed mechanism and VISAR's effectiveness to modern models, such as Qwen3.5, unclear.
  3. The motivation and visual-side intervention strategy closely overlap with Risk Awareness Injection (ICML 2026), yet the paper neither clearly distinguishes its contributions nor provides a direct comparison.
Questions for authors
  1. The paper only establishes correlation between safety‑neuron activations and refusal behavior, without direct causal validation. Could the authors perform targeted neuron interventions,such as suppressing safety neurons to test jailbreak susceptibility or boosting their activations to restore refusal behavior?
  2. The evaluation is limited to relatively outdated VLMs. Could the authors extend their experiments to more recent open-weight models, such as Qwen3.5, to verify whether the proposed safety-neuron suppression phenomenon and VISAR's effectiveness generalize to modern multimodal architectures?
  3. How does the proposed safety-neuron suppression mechanism differ from the risk signal dilution identified in Risk Awareness Injection (ICML 2026)? Given their closely related motivations and visual-side intervention strategies, could the authors clarify VISAR's novelty and provide a direct comparison with RAI?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for their positive assessment of our neuron-level mechanistic analysis, lightweight visual-side intervention design, and experimental results. In response to your questions, we have conducted additional experiments and further clarified our technical contributions.

W1 & Q1: Validating the Causal Effects of Safety Neurons

We designed a targeted neuron-suppression experiment to examine the effects of safety-neuron activations on the model’s safety behavior through direct intervention. Specifically, we selected the identified top-100 safety neurons in LLaVA-1.5-7B and set their activations to zero during inference to simulate impaired safety-neuron function. We then evaluated the model’s attack success rate (ASR) on HarmfulQA. The results are shown below.

ModelHarmfulQA ASR (%) ↓
LLaVA-1.5-7B18.79
+ Safety-neuron Suppression44.39

After suppressing the top-100 safety neurons, ASR increased substantially from 18.79% to 44.39%. This shows that targeted suppression of these activations alone can substantially weaken the model’s original refusal capability, with the inputs and all other model parameters held fixed. This intervention provides additional evidence of the functional role of safety neurons in refusal behavior, demonstrating that their activation states make a substantive contribution to maintaining the model’s refusal capability beyond their statistical association with refusal.

W2 & Q2: Generalization to the New-Generation Multimodal Model Qwen3.5

We further extended VISAR to Qwen3.5-9B and evaluated its defensive effectiveness on MM-SafetyBench to examine whether the proposed safety-state restoration mechanism transfers across multimodal architectures. The results are shown below.

Illegal ActivityMalware GenerationPornographyHate SpeechPhysical HarmFraudAvg. ↓
STS-TSTS-TSTS-TSTS-TSTS-TSTS-T
Qwen3.5-9B13.403.095.1520.4536.3654.5511.9311.9341.287.365.5212.8836.1116.6732.6410.397.7924.0319.53
+ VISAR (Ours)1.030.000.000.000.000.004.590.000.000.000.000.612.080.000.000.000.000.000.46

On Qwen3.5-9B, VISAR reduced the average ASR on MM-SafetyBench from 19.53% to 0.46%. This demonstrates that VISAR can generate corrective visual suffixes to guide a recent mainstream model toward restoring refusal-related internal safety states without updating pretrained model parameters. These results provide further evidence of the applicability of our proposed mechanism across model architectures and show that VISAR’s effectiveness extends beyond a particular model structure or training configuration.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3 & Q3: Differences from Risk Awareness Injection (RAI) and Novel Contributions

We thank the reviewer for highlighting the similarities between VISAR and RAI in their motivations and visual-side intervention strategies. We agree that both works address multimodal safety degradation induced by visual inputs and seek to improve safety through visual-side intervention. However, they differ substantively in their mechanistic findings, intervention targets, and implementation mechanisms.

(1) Mechanistic findings — risk signal dilution in representation space and safety-state suppression at the neuron level.

RAI examines cross-modal representation space and attributes multimodal safety degradation to Risk Signal Dilution caused by visual semantic noise. It characterizes the attenuation of risk information by analyzing the similarity between visual tokens and an unsafe prototype subspace.

VISAR instead examines functional units within the model and identifies Vision-induced Safety-neuron Suppression. We find that transferring harmful intent from text to the visual modality substantially weakens the activations of specific refusal-related neurons in the language backbone, shifting their activation distributions away from the refusal-aligned state induced by harmful text-only inputs. This suppression is particularly pronounced when harmful intent is primarily conveyed visually.

The distinction concerns what each work examines. RAI investigates whether risk semantics are sufficiently preserved in visual representations, whereas VISAR investigates whether visual inputs suppress the model’s existing refusal-related functional state. RAI explains safety degradation through risk-information representations, while VISAR examines its internal mechanism through neuron-level functional states.

(2) Intervention mechanisms — risk-semantic injection and restoration of refusal-related safety states.

RAI constructs an unsafe prototype subspace from predefined risk keywords and uses training-free sparse additive modulation to inject risk semantics into selected visual tokens at the model’s initial layer, strengthening risk signals in visual representations.

VISAR identifies refusal-related safety neurons by contrasting harmful and benign text inputs and constructs a differentiable safety-state probe, without relying on predefined risk keywords or risk-semantic prototypes. We then train a lightweight, plug-and-play SuffixViT to generate corrective visual suffixes under gradient supervision from the frozen probe. These suffixes guide the model toward restoring refusal-related neuron activations suppressed by visual inputs. Benign self-distillation further constrains the intervention’s impact on normal multimodal capabilities.

RAI therefore strengthens risk-semantic signals within visual representations, while VISAR uses visual-side intervention to restore suppressed refusal-related internal safety states. Although both intervene through the visual modality, their optimization objectives, supervision signals, and targeted internal mechanisms differ.

(3) Direct experimental comparison.

We agree that a direct comparison using the same models, datasets, and evaluation protocols would help clarify the performance differences between the methods. However, to the best of our knowledge, RAI has not yet provided a publicly available implementation suitable for reproduction, and its method involves specific representation-extraction and modulation procedures. Within the rebuttal period, we were unable to establish a sufficiently reliable and fair comparison protocol. We therefore refrain from making unverified claims about their relative performance.


We thank the reviewer again for their insightful and constructive comments. We will incorporate the additional experiments and analyses into the revised manuscript and add a systematic discussion of RAI to clarify the differences in research questions, mechanistic findings, and intervention mechanisms. We will also further articulate VISAR’s distinct contributions in identifying vision-induced safety-neuron suppression and using a differentiable safety-state probe to guide neuron-level safety-state restoration.

Thanks for the authors’ clarification. I find the experiments on Qwen3.5 and the principled comparison with RAI quite meaningful, which has addressed many of my concerns. However, I have a follow-up question regarding the first issue: currently, only the top-100 safety neurons are ablated, yielding a notable ASR increase on HarmfulQA. Critical control groups are still missing, leaving it unclear whether this effect stems from the specific safety-regulating role of these neurons, or merely represents generic model perturbation induced by suppressing an arbitrary set of neurons. Could you add a baseline with random suppression of 100 ordinary neurons (preferably with multiple random draws to report the ASR distribution), together with graded ablation controls, for instance, suppressing the top 10, 30, and 50 safety neurons separately to verify whether ASR rises in a dose-dependent manner and establish more robust causal evidence?

Reviewer_FctTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal safety degradation through neuron-level analysis and proposes VISAR, a visual-side intervention method that uses a differentiable safety-state probe to train a visual suffix generator. The method aims to restore refusal-associated neuron activations while preserving benign behavior, without modifying the pretrained VLM. Experiments across three VLMs and three jailbreak benchmarks show promising reductions in attack success rates, supported by component ablations and activation-level analyses.

Critical strengths

(1) The paper addresses a real weakness in multimodal safety: harmful intent can be conveyed through images, bypassing refusal behavior learned primarily through language. Improving this robustness without retraining the full VLM is practically valuable.

(2) The visual encoder, multimodal projector, and language backbone remain frozen. This reduces the risk of catastrophic changes to the base model, simplifies integration into existing systems, and makes the method attractive when model weights cannot be modified.

(3) The analysis examines intermediate MLP activations rather than treating jailbreak failures as purely input-output phenomena. The reported shift in refusal-associated neuron activations under multimodal inputs provides a useful hypothesis about how visual inputs affect safety behavior.

Critical weaknesses

(1) Benign utility evaluation is too narrow. ScienceQA-IMG provides useful evidence about visual question answering and scientific reasoning, but it is insufficient to support a broad claim that VISAR preserves benign multimodal utility. The current evaluation does not comprehensively establish whether the intervention affects: (I) General image understanding and detailed visual grounding. (II) OCR and reading of text embedded in images. (III) Fine-grained recognition and spatial reasoning. (IV) Complex multi-turn multimodal interactions. (V) Benign requests that resemble harmful instructions or discuss sensitive topics in legitimate contexts.

(2) Robustness against adaptive attacks is not evaluated. The experiments mainly compare VISAR with existing defenses on established benchmarks. They do not demonstrate robustness when an attacker knows that the defense appends learned visual suffix tokens. An adaptive attacker could potentially exploit the interaction between the original visual features and the suffix, or search for inputs that evade the probe's preferred activation pattern. The fact that the base VLM remains frozen does not by itself guarantee resistance to such attacks.

(3) The generalization of the learned visual suffix is insufficiently demonstrated. SuffixViT is trained using 2,000 harmful samples drawn from HADES and VLSBench and 1,000 benign image–caption pairs from Pick-a-Pic. Although the evaluation covers several benchmarks, the paper does not sufficiently establish how well the learned intervention generalizes beyond its training distribution. (I) Does the defense generalize to unseen harmful concepts, visual styles, and prompt templates? (II) Does it work on real photographs, screenshots, diagrams, and other visual inputs that differ from the training data? (III) How does it behave when harmful intent is only implicit or requires complex visual reasoning?

(4) The safety-neuron selection procedure has limited statistical validation. The method uses only 100 harmful text-only samples, retaining those that elicit explicit refusals, and 100 benign text-only questions to identify neurons. It selects the layer with the strongest separation and retains the top K=100 neurons. However, (I) Neuron selection may be sensitive to the particular samples and random seed. (II) No comprehensive stability analysis is reported for the selected neurons across different datasets or prompt variants.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for their positive assessment of VISAR’s practical value, visual-side intervention mechanism, and neuron-level analysis. We have conducted additional experiments to address the four key concerns and provide our responses below.

W1: Comprehensiveness of Benign Multimodal Utility Evaluation

We conducted five additional benign benchmark evaluations on InternVL-Chat-13B to assess VISAR’s ability to preserve utility across general visual understanding, text recognition, fine-grained visual perception, multimodal dialogue, and refusal tendencies on benign requests. Specifically, we evaluated MME, TextVQA, BLINK, ConvBench, and MOSSBench.

BenchmarkMME (Accuracy %, ↑)TextVQA (Accuracy %, ↑)BLINK (Accuracy %, ↑)ConvBench (Win Rate %, ↑)MOSSBench (Refusal Rate %, ↓)
InternVL76.7153.6340.880.870.33
+ CMRM67.6134.2120.050.521.00
+ VISAR (Ours)76.4552.6940.240.873.33

The results show that VISAR maintains performance close to that of the original model across all five evaluations. In comparison, CMRM, the strongest defense baseline, exhibits larger performance declines on the visual-understanding tasks above. These results suggest that, compared with defenses that may improve safety at the expense of normal multimodal capabilities, VISAR’s lightweight visual-side intervention better preserves the original model’s visual understanding and interaction capabilities while restoring its safety state.

W2: Robustness Against Adaptive Attacks

We constructed an adaptive attack targeting VISAR’s safety-state probe to evaluate its defensive capability when an attacker actively attempts to evade probe supervision. Specifically, we conducted experiments on LLaVA-1.5-7B using MM-SafetyBench, randomly selecting 100 harmful image–text pairs and applying PGD to optimize adversarial image perturbations. The attack explicitly uses feedback from the safety-state probe to optimize perturbations that evade its preferred refusal-aligned activation patterns, simulating an adaptive attacker with knowledge of VISAR’s internal defense mechanism. We then evaluated the attack success rate (ASR) of the original model and VISAR on the same attack instances.

MethodMM-SafetyBench ASR (%)Adaptive Attack ASR (%)
LLaVA-1.5-7B58.9968.00
+ VISAR (Ours)4.0213.00

The results show that even when the attacker uses probe feedback to actively optimize adversarial perturbations, VISAR limits ASR to 13.00%, substantially below the original model’s 68.00% under the adaptive attack. This result indicates that VISAR’s effectiveness extends beyond static adaptation to known attack patterns. Even when the attacker explicitly optimizes against its internal safety-state supervision signal, the visual-side intervention can still recover refusal behavior to some extent, demonstrating resilience against adaptive attacks.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3: Generalization of the Visual Suffix and Adaptation to Complex Visual Scenarios

To further assess VISAR’s ability to handle complex attack forms, we conducted additional evaluations based on the four visual attacks proposed in Jailbreaking Vision-Language Models Through the Visual Modality (ICML 2026). Specifically, we randomly selected 400 harmful text instructions and generated four categories of image–text attack instances following that work’s construction methods, covering Visual Cipher, Visual Object Replacement, Visual Text Replacement, and Visual Riddle. These attacks convey or conceal harmful intent through symbolic encoding, visual object replacement, text replacement within images, and visual analogy reasoning, respectively, placing different demands on the model’s cross-modal semantic understanding. We evaluated VISAR’s defensive effectiveness against these attacks on LLaVA-1.5-7B.

Attack TypeLLaVA-1.5-7B ASR (%)+ VISAR ASR (%)
Visual Cipher8616
Visual Object Replacement9326
Visual Text Replacement8829
Visual Riddle8430
Overall87.7525.25

The results show that VISAR provides consistent defensive benefits across all four visual attack categories, reducing average ASR from 87.75% to 25.25%. This provides further evidence of VISAR’s generalization across different forms of visual expression. The learned visual suffix transfers beyond specific training-time attack templates to attacks with different visual encoding mechanisms and reasoning requirements.

W4: Statistical Stability of Safety-Neuron Selection

We conducted an additional neuron-localization consistency experiment across different sample sets to assess sensitivity to sample selection. Specifically, we randomly constructed three mutually disjoint datasets, A, B, and C, from AdvBench and Alpaca, each containing 100 harmful samples and 100 benign samples. We then independently performed safety-neuron localization using the same model and localization configuration, and compared the resulting neuron-importance rankings across datasets. We used Spearman’s rank correlation coefficient ρ to measure consistency across sample sets.

Dataset PairA-BB-CA-C
ρ0.8140.7670.772

All three pairwise correlations exceed 0.75, indicating strong consistency in the neuron-importance rankings obtained from different sample sets. This suggests that the safety-neuron localization underlying VISAR is not an artifact of a particular sample set, but exhibits relatively stable structural patterns across different random partitions. These results provide additional statistical support for the subsequent construction of the safety-state probe.


We thank the reviewer again for their constructive suggestions. In response, we have added five benign utility benchmark evaluations, a probe-targeted adaptive attack experiment, evaluations of four new visual attack categories, and an analysis of safety-neuron localization stability across different sample sets. These results further strengthen the evaluation of VISAR across utility preservation, adversarial robustness, generalization across attack forms, and mechanistic reliability. We will incorporate these experimental results, evaluation settings, and analyses into the revised manuscript, and further clarify the method’s current scope and limitations. We again thank the reviewer for their careful review and valuable suggestions.

Thanks for the authors' detailed response. It addresses most of my concerns. Recent work OO-VTP [ref1] enhances model robustness via visual token pruning, i.e., prunes the visual tokens that suppress MLP neurons associated with harmful-request refusal. Thus, it is closely related to VISAR, as VISAR corrects the MLP neurons suppressed by visual tokens. OO-VTP shows that pruning the misaligned visual tokens leads to robustness against jailbreaks and hallucinations. Could the authors investigate whether VISAR also mitigates hallucinations? Such an evaluation would help clarify whether its benefits extend to visual grounding beyond restoring refusal behavior.

[ref1] Visual Token Compression Enhances Robustness of MLLMs, ACM Multimedia 2026.

Reviewer_js9uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies vision-induced safety-neuron suppression in VLMs. It proposes VISAR to restore refusal-aligned activation patterns. VISAR identifies refusal-related MLP neurons and uses a differentiable safety probe to train SuffixViT, which generates corrective visual suffix tokens. Training uses safety-state restoration and benign self-distillation while keeping the pretrained VLM and probe frozen. Experiments on three VLM backbones report substantial reductions in attack success rate across multimodal jailbreak settings, with relatively small changes in ScienceQA-IMG accuracy.

Critical strengths
  1. VISAR uses a differentiable safety-neuron probe to train SuffixViT while keeping the pretrained VLM frozen. This provides a concrete way to use internal activations to guide visual-side intervention.
  2. VISAR has the lowest reported average MM-SafetyBench ASR among the compared defenses on all three VLM backbones. Ablations on LLaVA support the contributions of visual suffixes, the restoration objective, and safety-neuron selection.
Critical weaknesses
  1. The refusal-aligned safety state needs stronger behavioral validation. Neuron selection contrasts refused harmful text with answered benign text, and the probe uses condition-based labels. Held-out validation against actual refusal behavior is not reported, leaving the interpretation of safety-state restoration uncertain.
  2. The evidence for preserving benign utility is limited to ScienceQA-IMG accuracy. Without direct false-refusal evaluation on legitimate open-ended requests, it remains unclear whether VISAR improves safety selectively rather than increasing refusal rates on benign requests.
Questions for authors
  1. How were harmful multimodal probe negatives selected, and were they verified to elicit non-refusal responses from the target model?
  2. How well does the safety probe predict actual refusal behavior on multimodal samples held out from both neuron selection and probe training?
  3. Beyond ScienceQA-IMG, do you have false-refusal measurements on legitimate open-ended requests and additional evaluations of benign visual understanding?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for recognizing VISAR’s methodological design and experimental results. We take the reviewer’s suggestions regarding the behavioral validity of the safety state and the adequacy of benign utility evaluation very seriously. To address these concerns, we conducted additional experiments on probe generalization and false refusals on benign requests, and provide our point-by-point responses below.

Q1:Negative Sample Selection and Behavioral Validation

According to Algorithm 1, the probe’s negative set D−\mathcal{D}^{-} consists of three components, harmful images paired with benign text Dhb\mathcal{D}{hb}, benign text-only data Db\mathcal{D}{b}, and benign images paired with benign text Dbb\mathcal{D}{bb}. The corresponding positive set D+\mathcal{D}^{+} comprises harmful text-only inputs Dh\mathcal{D}{h} that elicit explicit refusal responses from the model.

Among these, Dhb\mathcal{D}_{hb} is a particularly important negative subset. These samples are constructed by converting the original harmful instructions into visual content and pairing them with benign textual prompts. On the undefended target VLM, these inputs can exploit the visual modality to bypass the original text-based safety alignment and elicit harmful, non-refusal responses. Their negative labels therefore reflect the target model’s actual response behavior, rather than being determined solely by whether the input contains harmful content.

Specifically, we first select harmful text-only samples that reliably elicit explicit refusals as positive examples, and then construct negative examples through the corresponding multimodal transformations. For harmful multimodal samples, we automatically check the target model’s actual outputs and manually review a subset to confirm non-refusal behavior. Benign samples correspond to normal, non-refusal responses.

The probe thus learns differences in internal activations associated with refusal and non-refusal behavior, rather than simply classifying the semantic content of the input. This design uses the model’s own behavior as supervision and provides a more clearly defined target for subsequently restoring the refusal-aligned safety-neuron state.

Q2:Generalization of the Safety Probe to Actual Refusal Behavior

We conducted an independent held-out generalization experiment on LLaVA-1.5-7B to assess the consistency between probe predictions and the model’s actual response behavior. Specifically, we randomly sampled 100 inputs from each of four evaluation datasets, yielding 400 test samples in total. These cover harmful multimodal inputs from MM-SafetyBench, benign visual-understanding requests from MME, benign text and image–text requests from SimpleQA, and harmful text requests from HarmfulQA. None of these test samples were used for safety-neuron selection or probe training. We first determined behavioral labels from the target model’s actual responses to the original inputs, and then independently predicted these labels using the frozen safety probe applied to the corresponding neuron activations.

A predicted label of 1 indicates that the model enters the refusal-aligned safety-neuron activation state, while a label of 0 indicates that it does not enter this state. The results are shown below.

BenchmarkPredicted as 1 (Refusal-Aligned State)Predicted as 0 (Non-Refusal State)ACC
MM-SafetyBench19999%
MME0100100%
SimpleQA0100100%
HarmfulQA1000100%
Average——99.75%

The safety probe achieved an average accuracy of 99.75% on fully independent test samples. These results provide behavioral validation of the strong correspondence between safety-neuron activation patterns and the model’s refusal behavior. They also indicate that the probe generalizes beyond the activation distributions of its training samples to unseen multimodal inputs, further supporting VISAR’s use of the probe to reliably guide safety-state restoration.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q3:False-Refusal Rate and Visual Understanding on Benign Requests

To further assess VISAR’s impact on normal model behavior, we conducted additional experiments on MME and MOSSBench. MME evaluates accuracy on benign visual-understanding tasks, while MOSSBench evaluates refusal rates on legitimate open-ended requests to measure potential false refusals.

The results on InternVL-Chat-13B are shown below.

ModelMME Accuracy (%) ↑MOSSBench Refusal Rate (%) ↓
InternVL-Chat-13B76.710.33
+ VISAR (Ours)76.453.33

After introducing VISAR, MME accuracy decreased by only 0.26 percentage points, indicating that the visual safety intervention did not substantially impair the model’s original visual-understanding ability. Meanwhile, VISAR’s refusal rate on legitimate open-ended requests in MOSSBench was 3.33%, which remained low overall. This suggests that its safety improvements were achieved without broadly increasing the model’s tendency to refuse requests.

This result is consistent with VISAR’s design objective. We use the safety probe to guide visual suffixes toward restoring refusal-related neuron activations suppressed by harmful multimodal inputs, while benign self-distillation constrains the intervention’s impact on normal model behavior. The pretrained model parameters remain frozen, and training does not require the model to generate predefined refusal responses. VISAR therefore aims to restore a targeted refusal-aligned state rather than simply increase the model’s overall refusal tendency.

We thank the reviewer again for their constructive suggestions. These comments helped us clarify the relationship between safety-neuron activations and actual refusal behavior, and motivated additional utility evaluations covering both open-ended benign requests and visual understanding. We will incorporate these experimental results and analyses into the revised manuscript to strengthen the behavioral validity of the method and the credibility of our experimental conclusions.

Thank you for the detailed response. The additional utility evaluations largely address my second concern. My remaining concern is the probe’s behavioral validation. Most positive probe predictions in the held-out evaluation come from text-only HarmfulQA, and it is unclear how many multimodal samples actually elicited refusals. Could you report actual refusal counts and a probe confusion matrix for held-out harmful multimodal samples, covering both refused and non-refused responses? This would clarify whether the probe reliably predicts refusal behavior in multimodal settings.

AuthorsOpenReview ↗

Response to Reviewer js9u

We sincerely thank the reviewer for the positive and constructive discussion. Your recognition of our work and increased score have been very encouraging. We are pleased to have the opportunity to further address your concerns. To this end, we additionally examined the actual refusal behavior on harmful multimodal samples. Specifically, among the 100 harmful multimodal samples from MM-SafetyBench used in Q2, the model explicitly refused 41 samples, while the remaining 59 resulted in successful jailbreaks, yielding the following confusion matrix.

Actual BehaviorProbe Predicts 0Probe Predicts 1
Successful Jailbreak (Non-refusal)581
Successful Refusal410

These results are consistent with our expectations. As described in Section 4.2, the purpose of our probe is not to train a binary classifier that accurately predicts refusal behavior, but to characterize whether the safety neurons enter the activation state induced by harmful text-only inputs (the refusal state). Specifically, we use harmful text-only inputs that elicit explicit refusals as positive references (label 1), and harmful multimodal inputs and benign inputs as negative examples (label 0). Therefore, the probe’s output reflects how closely the current safety-neuron activation pattern resembles the text-only refusal reference state, rather than directly classifying the final generated behavior.

This design is consistent with VISAR’s objective. VISAR does not rely on the probe to accurately predict refusal behavior. Instead, it uses the probe as a differentiable supervision signal for the internal state. By minimizing the restoration loss, VISAR guides SuffixViT to restore the safety-neuron activation patterns of harmful multimodal inputs toward those induced by harmful text-only inputs (the refusal state).

We thank the reviewer for prompting us to clarify this distinction. In the revised manuscript, we will further clarify the probe’s design objective to avoid conflating internal safety-state estimation with prediction of the final refusal behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.