- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies safety degradation in large vision-language models (LVLMs) under benign fine-tuning, showing that updating only the vision encoder while freezing the language backbone can substantially increase attack success rates. The authors propose the vision-sensitivity hypothesis and analyze update-induced responses and harmful–benign separation along a fixed refusal direction. They further introduce SAVR to constrain representation drift along this direction induced by vision encoder updates. Experiments across three models and three downstream tasks show that SAVR mitigates safety degradation under V-tuning while generally preserving downstream utility.
Critical strengths
- By isolating vision encoder updates, the paper shows that freezing the language backbone does not guarantee unchanged safety behavior of the overall model, addressing a meaningful research question.
- Experiments cover three models, three downstream tasks, and multiple fine-tuning modes, reporting both safety and utility and providing substantial empirical support for the main observation.
- The refusal direction is selected using intervention-based criteria, and the analysis further separates parameter update size from response per unit update, providing a useful basis for the representation-level discussion.
Critical weaknesses
-
The practical need to update the vision encoder and the scope of SAVR remain unclear. V-tuning helps isolate the effects of vision encoder updates, but does not by itself establish that commonly used fine-tuning workflows require additional vision regularization. The paper does not sufficiently clarify which tasks require updating the vision encoder or whether SAVR offers a better safety–utility trade-off than approaches that keep it frozen.
-
Higher refusal sensitivity does not sufficiently establish a vulnerability specific to the vision encoder. Equation (7) incorporates both parameter update size and the resulting representation response, while Appendix C indicates that the higher values are mainly driven by larger vision encoder updates. Using the same learning rate across modules does not imply comparable perturbations. The results therefore do not adequately distinguish the vision encoder’s intrinsic sensitivity from the effects of the chosen optimization configuration.
-
The relationship between refusal margin shrinkage and actual safety degradation is not sufficiently established. In Table 2, V-tuning causes the largest margin reduction in every setting, yet it does not consistently yield the highest ASR in Table 1. For example, L-tuning produces higher ASRs on all three tasks for Llama-3.2-Vision. This metric is therefore insufficient to explain the extent of safety degradation across fine-tuning modes.
-
The benefits of SAVR have not been sufficiently distinguished from those of general update constraints. Equation (10) constrains representation drift relative to the aligned model, but the comparisons mainly involve unprotected fine-tuning and safety-data fine-tuning. It remains unclear whether the gains depend on the selected refusal direction or could also be achieved by reducing vision encoder updates or preserving representations more generally. Targeted controls are needed to establish the specific value of this design.
Questions for authors
-
How should the regularization under full-tuning be interpreted? Equation (10) measures the overall representation drift, including changes induced by language backbone updates, whereas Equation (11) applies the regularization gradient only to the vision encoder. Could this design cause the vision encoder to compensate for changes induced by the language backbone?
-
How consistently does the fixed refusal direction remain valid after fine-tuning? Appendix B mentions post-fine-tuning bypass ablations against a random-direction control. Could the authors provide quantitative results across models and fine-tuning modes to show that the original direction remains a reliable indicator of refusal behavior?