- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies vision-induced safety-neuron suppression in VLMs, where visual inputs weaken refusal-related neuron activations and increase vulnerability to multimodal jailbreaks. This paper proposes VISAR, which uses a learned visual suffix guided by a safety-neuron probe to restore refusal-aligned activations without modifying the pretrained VLM.
Critical strengths
- This work provides a neuron-level perspective on multimodal safety degradation, revealing how visual inputs affect refusal-related activations.
- VISAR restores safety-related activations through visual suffix tokens without modifying pretrained VLM parameters, introducing minimal inference overhead.
- Experiments across three VLMs and multiple jailbreak benchmarks demonstrate substantial reductions in attack success rates while largely preserving benign performance.
Critical weaknesses
- This work only establishes a correlation between safety-neuron activations and refusal behavior, without direct neuron-level interventions to validate their causal role.
- The experiments focus on relatively outdated VLMs, leaving the generalizability of the proposed mechanism and VISAR's effectiveness to modern models, such as Qwen3.5, unclear.
- The motivation and visual-side intervention strategy closely overlap with Risk Awareness Injection (ICML 2026), yet the paper neither clearly distinguishes its contributions nor provides a direct comparison.
Questions for authors
- The paper only establishes correlation between safety‑neuron activations and refusal behavior, without direct causal validation. Could the authors perform targeted neuron interventions,such as suppressing safety neurons to test jailbreak susceptibility or boosting their activations to restore refusal behavior?
- The evaluation is limited to relatively outdated VLMs. Could the authors extend their experiments to more recent open-weight models, such as Qwen3.5, to verify whether the proposed safety-neuron suppression phenomenon and VISAR's effectiveness generalize to modern multimodal architectures?
- How does the proposed safety-neuron suppression mechanism differ from the risk signal dilution identified in Risk Awareness Injection (ICML 2026)? Given their closely related motivations and visual-side intervention strategies, could the authors clarify VISAR's novelty and provide a direct comparison with RAI?
5 comments
Official Comment by Authors (2/2)
W3 & Q3: Differences from Risk Awareness Injection (RAI) and Novel Contributions
We thank the reviewer for highlighting the similarities between VISAR and RAI in their motivations and visual-side intervention strategies. We agree that both works address multimodal safety degradation induced by visual inputs and seek to improve safety through visual-side intervention. However, they differ substantively in their mechanistic findings, intervention targets, and implementation mechanisms.
(1) Mechanistic findings — risk signal dilution in representation space and safety-state suppression at the neuron level.
RAI examines cross-modal representation space and attributes multimodal safety degradation to Risk Signal Dilution caused by visual semantic noise. It characterizes the attenuation of risk information by analyzing the similarity between visual tokens and an unsafe prototype subspace.
VISAR instead examines functional units within the model and identifies Vision-induced Safety-neuron Suppression. We find that transferring harmful intent from text to the visual modality substantially weakens the activations of specific refusal-related neurons in the language backbone, shifting their activation distributions away from the refusal-aligned state induced by harmful text-only inputs. This suppression is particularly pronounced when harmful intent is primarily conveyed visually.
The distinction concerns what each work examines. RAI investigates whether risk semantics are sufficiently preserved in visual representations, whereas VISAR investigates whether visual inputs suppress the model’s existing refusal-related functional state. RAI explains safety degradation through risk-information representations, while VISAR examines its internal mechanism through neuron-level functional states.
(2) Intervention mechanisms — risk-semantic injection and restoration of refusal-related safety states.
RAI constructs an unsafe prototype subspace from predefined risk keywords and uses training-free sparse additive modulation to inject risk semantics into selected visual tokens at the model’s initial layer, strengthening risk signals in visual representations.
VISAR identifies refusal-related safety neurons by contrasting harmful and benign text inputs and constructs a differentiable safety-state probe, without relying on predefined risk keywords or risk-semantic prototypes. We then train a lightweight, plug-and-play SuffixViT to generate corrective visual suffixes under gradient supervision from the frozen probe. These suffixes guide the model toward restoring refusal-related neuron activations suppressed by visual inputs. Benign self-distillation further constrains the intervention’s impact on normal multimodal capabilities.
RAI therefore strengthens risk-semantic signals within visual representations, while VISAR uses visual-side intervention to restore suppressed refusal-related internal safety states. Although both intervene through the visual modality, their optimization objectives, supervision signals, and targeted internal mechanisms differ.
(3) Direct experimental comparison.
We agree that a direct comparison using the same models, datasets, and evaluation protocols would help clarify the performance differences between the methods. However, to the best of our knowledge, RAI has not yet provided a publicly available implementation suitable for reproduction, and its method involves specific representation-extraction and modulation procedures. Within the rebuttal period, we were unable to establish a sufficiently reliable and fair comparison protocol. We therefore refrain from making unverified claims about their relative performance.
We thank the reviewer again for their insightful and constructive comments. We will incorporate the additional experiments and analyses into the revised manuscript and add a systematic discussion of RAI to clarify the differences in research questions, mechanistic findings, and intervention mechanisms. We will also further articulate VISAR’s distinct contributions in identifying vision-induced safety-neuron suppression and using a differentiable safety-state probe to guide neuron-level safety-state restoration.
Thanks for the authors’ clarification. I find the experiments on Qwen3.5 and the principled comparison with RAI quite meaningful, which has addressed many of my concerns. However, I have a follow-up question regarding the first issue: currently, only the top-100 safety neurons are ablated, yielding a notable ASR increase on HarmfulQA. Critical control groups are still missing, leaving it unclear whether this effect stems from the specific safety-regulating role of these neurons, or merely represents generic model perturbation induced by suppressing an arbitrary set of neurons. Could you add a baseline with random suppression of 100 ordinary neurons (preferably with multiple random draws to report the ASR distribution), together with graded ablation controls, for instance, suppressing the top 10, 30, and 50 safety neurons separately to verify whether ASR rises in a dose-dependent manner and establish more robust causal evidence?
Official Comment by Authors (1/2)
We thank the reviewer for their positive assessment of our neuron-level mechanistic analysis, lightweight visual-side intervention design, and experimental results. In response to your questions, we have conducted additional experiments and further clarified our technical contributions.
W1 & Q1: Validating the Causal Effects of Safety Neurons
We designed a targeted neuron-suppression experiment to examine the effects of safety-neuron activations on the model’s safety behavior through direct intervention. Specifically, we selected the identified top-100 safety neurons in LLaVA-1.5-7B and set their activations to zero during inference to simulate impaired safety-neuron function. We then evaluated the model’s attack success rate (ASR) on HarmfulQA. The results are shown below.
After suppressing the top-100 safety neurons, ASR increased substantially from 18.79% to 44.39%. This shows that targeted suppression of these activations alone can substantially weaken the model’s original refusal capability, with the inputs and all other model parameters held fixed. This intervention provides additional evidence of the functional role of safety neurons in refusal behavior, demonstrating that their activation states make a substantive contribution to maintaining the model’s refusal capability beyond their statistical association with refusal.
W2 & Q2: Generalization to the New-Generation Multimodal Model Qwen3.5
We further extended VISAR to Qwen3.5-9B and evaluated its defensive effectiveness on MM-SafetyBench to examine whether the proposed safety-state restoration mechanism transfers across multimodal architectures. The results are shown below.
On Qwen3.5-9B, VISAR reduced the average ASR on MM-SafetyBench from 19.53% to 0.46%. This demonstrates that VISAR can generate corrective visual suffixes to guide a recent mainstream model toward restoring refusal-related internal safety states without updating pretrained model parameters. These results provide further evidence of the applicability of our proposed mechanism across model architectures and show that VISAR’s effectiveness extends beyond a particular model structure or training configuration.