- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies vision-induced safety degradation (VD) in multimodal large language models, where a model that safely refuses a harmful text-only request may comply when an image is added. The authors argue that existing representation-level interventions rely on fixed schemes that fail to adapt as generation evolves. They propose TrajBridge, an inference-time framework with three components: offline calibration of layer-wise correction references derived from native safe/unsafe response pairs, online dynamic localization of intervention sites via probe-based evaluation, and a correction controller that governs when intervention activates, releases, and reactivates. Experiments on two MLLM backbones across multiple safety benchmarks demonstrate improved safety recovery while largely preserving model capabilities.
Critical strengths
S1. Clear problem formulation with practical significance.
Vision-induced safety degradation (VD) is a real and observable failure mode in deployed MLLMs.
S2. Well-motivated method with clear presentation and rigorous ablations.
The proposed TrajBridge is conceptually clean and easy to follow. The authors do a commendable job walking through the design intuition.
Critical weaknesses
W1. The VD phenomenon itself is barely studied, and there is a mismatch between the paper's motivation and its actual contribution.
While the paper motivates TrajBridge through the lens of VD, it contributes little to our mechanistic understanding of why VD occurs. The actual proposal is a general-purpose inference-time dynamic safety control framework, with VD serving as an application context.
The paper would be strengthened by either (a) providing deeper analysis of VD's underlying causes, or (b) reframing the contribution as a general dynamic control method rather than a VD-specific solution.
W2. The assumption that text-only behavior can serve as a safe reference breaks down in several realistic scenarios.
The offline calibration step requires collecting VD pairs , where is a safe text-only response. This implicitly assumes that text-only behavior provides a valid safety reference. However, this assumption fails in several important cases:
(a) Image-as-payload attacks: In benchmarks like FigStep, the image itself contains the harmful instruction (for example, text rendered in the image reading "how to make a bomb").
(b) Image-grounded semantic ambiguity: For instructions like "How do I do this?", the text alone is ambiguous without the image. The text-only "safe" response may be an unhelpful refusal such as "I need more context."
W3. The cost of obtaining calibration data and the risk of distribution shift are not addressed.
The paper reports a modest calibration set size (64 VD pairs + 32 stable-safe pairs + 48 benign inputs), which appears lightweight. However, the hidden cost of obtaining these VD pairs is ignored:
(a) VD pair acquisition requires expensive pre-scanning: One must first run the base model on all candidate pairs (once for text-only, once for multimodal), then apply a safety judge to identify the VD subset. Appendix F shows that VD rates vary dramatically across benchmarks (VL-Safe 57% vs. HoliSafe SSS 0.15%). In low-VD-rate settings, one would need to scan tens of thousands of candidates to obtain just 64 pairs.
(b) Distribution shift between calibration and deployment: The calibration VD pairs are drawn from existing benchmarks (VL-Safe, SIUO, MSTS). Deployed systems, however, will likely face very different VD patterns, including novel jailbreak strategies, new image distributions, or new task domains.
Questions for authors
(1) In a deployed setting, who is responsible for constructing the VD pairs? Does this effectively require running a full VD audit before deployment?
(2) Have you evaluated whether the reference directions generalize to unseen distributions?