acceptodds
Withdrawn from ICLR 2027

Vision Encoder Is a Hidden Culprit for Safety Degradation in Benign LVLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benign fine-tuning can undermine the safety alignment of Large Vision-Language Models (LVLMs), a risk commonly associated with their language counterpart. We reveal that the vision encoder is also unsafe or less safe than the language backbone. By updating only the vision encoder (V-tuning), substantial safety degradation is observed across various LVLMs and downstream tasks. Such safety deterioration matches or even exceeds that of language-backbone or full-model tuning. To explain this vulnerability, we propose the vision-sensitivity hypothesis and analyze how parameter updates perturb representations along a fixed refusal direction. Our first-order analysis finds that the refusal-mediating structure in LVLMs is significantly more sensitive to the vision encoder updates. Based on this, we then demonstrate that V-tuning substantially reduces the separation between harmful and benign representations along the refusal direction more than the other tuning modes. Motivated by these findings, we introduce SAVR, Sensitivity-Aware Vision Regularization, to preserve the LVLM's refusal structure during vision encoder fine-tuning. SAVR successfully reduces attack success rates by up to 98% relative to unprotected V-tuning while preserving utility. It also retains most of the aligned model's harmful-benign separation. We believe that this work offers new insights for further studies to understand and address fine-tuning risks in LVLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision Encoder Vulnerability, Large Vision-Language Model, Benign Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XSbUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies safety degradation in large vision-language models (LVLMs) under benign fine-tuning, showing that updating only the vision encoder while freezing the language backbone can substantially increase attack success rates. The authors propose the vision-sensitivity hypothesis and analyze update-induced responses and harmful–benign separation along a fixed refusal direction. They further introduce SAVR to constrain representation drift along this direction induced by vision encoder updates. Experiments across three models and three downstream tasks show that SAVR mitigates safety degradation under V-tuning while generally preserving downstream utility.

Critical strengths
  • By isolating vision encoder updates, the paper shows that freezing the language backbone does not guarantee unchanged safety behavior of the overall model, addressing a meaningful research question.
  • Experiments cover three models, three downstream tasks, and multiple fine-tuning modes, reporting both safety and utility and providing substantial empirical support for the main observation.
  • The refusal direction is selected using intervention-based criteria, and the analysis further separates parameter update size from response per unit update, providing a useful basis for the representation-level discussion.
Critical weaknesses
  1. The practical need to update the vision encoder and the scope of SAVR remain unclear. V-tuning helps isolate the effects of vision encoder updates, but does not by itself establish that commonly used fine-tuning workflows require additional vision regularization. The paper does not sufficiently clarify which tasks require updating the vision encoder or whether SAVR offers a better safety–utility trade-off than approaches that keep it frozen.

  2. Higher refusal sensitivity does not sufficiently establish a vulnerability specific to the vision encoder. Equation (7) incorporates both parameter update size and the resulting representation response, while Appendix C indicates that the higher values are mainly driven by larger vision encoder updates. Using the same learning rate across modules does not imply comparable perturbations. The results therefore do not adequately distinguish the vision encoder’s intrinsic sensitivity from the effects of the chosen optimization configuration.

  3. The relationship between refusal margin shrinkage and actual safety degradation is not sufficiently established. In Table 2, V-tuning causes the largest margin reduction in every setting, yet it does not consistently yield the highest ASR in Table 1. For example, L-tuning produces higher ASRs on all three tasks for Llama-3.2-Vision. This metric is therefore insufficient to explain the extent of safety degradation across fine-tuning modes.

  4. The benefits of SAVR have not been sufficiently distinguished from those of general update constraints. Equation (10) constrains representation drift relative to the aligned model, but the comparisons mainly involve unprotected fine-tuning and safety-data fine-tuning. It remains unclear whether the gains depend on the selected refusal direction or could also be achieved by reducing vision encoder updates or preserving representations more generally. Targeted controls are needed to establish the specific value of this design.

Questions for authors
  1. How should the regularization under full-tuning be interpreted? Equation (10) measures the overall representation drift, including changes induced by language backbone updates, whereas Equation (11) applies the regularization gradient only to the vision encoder. Could this design cause the vision encoder to compensate for changes induced by the language backbone?

  2. How consistently does the fixed refusal direction remain valid after fine-tuning? Appendix B mentions post-fine-tuning bypass ablations against a random-direction control. Could the authors provide quantitative results across models and fine-tuning modes to show that the original direction remains a reliable indicator of refusal behavior?

Reviewer_6176OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers the vision encoder as the primary reason for safety degradation during LVLM fine-tuning on benign data. It proposes the vision-sensitivity hypothesis, which argues that fine-tuning only the vision encoder (V-tuning) shrinks the harmful-benign refusal margin. To mitigate this issue, the authors propose a method called SAVR, which penalizes the drift of activation along the refusal direction.

Critical strengths
  1. The experiments are controlled and the motivation experiments are interesting.
  2. The method is simple and elegant.
  3. The presentation is clear.
Critical weaknesses
  1. In Table 2, full-tuning yields a refusal margin similar to L-tuning across all settings. Table 1 reflects the same pattern, with full-tuning ASR staying similar with L-tuning. If encoder updates were indeed a primary reason of safety degradation, updating the encoder alongside the backbone should makes the full-tuning ASR much worse.

  2. Relying on a single safety benchmark (HADES) provides a narrow evaluation of safety alignment. Moreover, SAVR is evaluated only against safety fine-tuning baselines. More safety benchmarks, utility benchmarks, and safety methods should be considered.

Questions for authors
  1. In Table 5, L-tuning consistently yields better utility than V-tuning: why not simply freeze the vision encoder? Furthermore, V-tuning barely improves utility (InternVL3-8B and Llama-3.2), and sometimes actively hurts it, while causing ASR to spike dramatically (e.g., Qwen3-VL on M3CoT drops utility from 62.1 to 56.8 while ASR jumps from 2.3% to 55.8%; InternVL3 on M3CoT shows a marginal utility gain from 51.6 to 52.4 alongside an ASR surge from 23.6% to 73.3%). Could this indicate that the learning rate for the encoder is set too high, causing representation collapse? Per-block learning rates are currently not reported.

  2. Is the probing setup structurally biased toward the encoder? Because sensitivity and refusal margins are evaluated at an intermediate backbone layer, any downstream representation changes introduced by L-tuning past that point are completely missed. In contrast, every encoder update flows directly through the probe, which seems to inherently favor the encoder by construction.

  3. Comparing Table 2 with Table 1: L-tuning retains a much larger margin than V-tuning yet still exhibits a higher ASR (e.g., InternVL3 on RefCOCO maintains a margin of 54.48 vs. 33.15, yet has an ASR of 57.7% vs. 48.3%). Does this undermine the claimed connection between safety margin and safety degradation?

  4. Under V-tuning, any regularizer that keeps the encoder close to its original state should naturally suppress ASR, making it unclear what value SAVR’s specific refusal-direction mechanism actually matters.

  5. In Tables 1 and 3, RefCOCO is the only task where full-tuning (as well as L-tuning) yields a higher ASR than V-tuning across all three models (e.g., Qwen3-VL: 49.9% vs. 27.7%). Yet, Table 2 shows V-tuning shrinking the refusal margin far more severely on RefCOCO (Qwen3-VL drops to 9.32 under V-tuning vs. 35.14 under full-tuning). What accounts for this reversal?

  6. Novelty: The vision encoder’s role in LVLM safety is not new. Robust CLIP [1], Sim-CLIP [2] already show that fine-tuning or hardening the encoder improves robustness against adversarial and jailbreak attacks. Prior work cited by the paper (e.g., [3]) also attributes LVLM safety degradation to the distribution shift introduced by visual inputs. While the specific benign fine-tuning setting may be new, the claim to be “the first to identify the vision encoder as a fine-tuning vulnerability” seems overstated.

[1] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. In ICML, 2024.

[2] Md Zarif Hossain and Ahmed Imteaj. Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models. arXiv:2407.14971, 2024.

[3] Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, and Yassine Benajiba. Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models. In Findings of ACL, 2025.

Reviewer_QMaDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the degradation of safety alignment in Large Vision-Language Models (LVLMs) during fine-tuning on benign data, with a specific focus on the vision encoder—a component relatively overlooked in prior research. Through a fine-tuning approach that updates only the vision encoder (termed "V-tuning"), the authors demonstrate that the model's attack success rate can increase significantly, even when the language model backbone remains frozen.

To explain this phenomenon, the authors propose the "Vision-Sensitivity Hypothesis," which analyzes how the model's hidden representations shift along a fixed "refusal direction"—a concept used to characterize the relationship between vision encoder updates and the erosion of safety alignment. Building on this, the authors introduce "Sensitivity-Aware Vision Regularization" (SAVR), a method that mitigates the decline in safety alignment by constraining changes in representations along the refusal direction during the fine-tuning process.

Critical strengths
  1. Technical novelty and innovation The paper offers a degree of innovation by focusing specifically on the visual encoder as the subject of study regarding safety degradation.

  2. Experimental rigor and validation The paper covers three types of LVLM architectures, three categories of downstream tasks, and various parameter fine-tuning strategies.

  3. Clarity of presentation The overall structure of the paper is clear: it begins by examining observed experimental phenomena, proceeds to a sensitivity analysis, and subsequently develops defense methods based on these findings.

Critical weaknesses
  1. Experimental Evaluation Table 1 provides compelling evidence that fine-tuning the visual encoder can significantly impact safety performance. For instance, after applying V-tuning to Qwen3-VL on the M3CoT task, the Attack Success Rate (ASR) increased from 2.3% to 55.8%. However, the conclusion that "visual encoders are more prone to causing safety degradation than language backbones" does not hold true across all experimental configurations. For example, in the RefCOCO experiments with Llama-3.2-Vision, V-tuning resulted in an ASR of 55.7%, whereas L-tuning reached 78.9%. Furthermore, the effectiveness of the SAVR defense varies significantly across models. In the case of InternVL3 on M3CoT, the ASR decreased from 73.3% to 43.6%, yet a substantial residual attack success rate remains. In some full-model fine-tuning scenarios, the improvement provided by SAVR is also relatively limited. It is recommended that the authors include comparisons based on the norm of parameter updates, sensitivity analysis regarding learning rates, experiments using multiple random seeds, and a broader range of safety evaluation benchmarks.

  2. Comparison with Related Work The most notable innovation of this paper is the identification of safety degradation resulting from the independent fine-tuning of visual encoders. In terms of its mathematical formulation, SAVR is similar to existing "activation-preservation" methods; its primary distinction lies in the selection of the "refusal direction" as the target for regularization. The authors should demonstrate that this specific direction holds unique significance for safety. For example, they could conduct control experiments using random directions, principal component directions, or other task-relevant directions.

  3. Discussion of Broader Impact and Significance The paper suggests that the safety of multimodal models must be considered in the context of the entire model architecture, rather than focusing solely on the language backbone. This insight offers valuable implications for modular fine-tuning and the deployment of multimodal models. However, as the current results are based on specific models, fine-tuning tasks, and safety benchmarks, they are insufficient to support the broad conclusion that visual encoders are inherently more vulnerable in terms of safety.

Questions for authors

How does the author account for factors such as the magnitude of parameter updates, the learning rate, and the optimizer state when comparing visual sensitivity?

Is it possible to demonstrate a statistically significant correlation between visual sensitivity, changes in refusal intervals, and the Attack Success Rate (ASR)?

How accurate is the first-order Jacobian approximation in predicting outcomes during the actual fine-tuning process?

How are the specific layers and token positions corresponding to the "refusal direction" determined? Do changes to these settings affect the conclusions?

Does SAVR offer a consistent advantage over general-purpose methods like feature distillation, activation matching, and weight regularization?

Is the regularization coefficient for SAVR selected using an independent validation set?

Can SAVR defend against purely text-based harmful requests as well as other types of multimodal attacks?

Why is the improvement provided by SAVR notably smaller under certain full-model fine-tuning configurations?

What are the additional costs introduced by SAVR in terms of training time, GPU memory usage, and activation caching?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.