acceptodds
Withdrawn from ICLR 2027

TrajBridge: Trajectory-Guided Safety Recovery For Multimodal Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) extend language models with visual understanding, but the visual inputs can also disrupt safety behavior that is otherwise preserved under text-only input. In particular, the same MLLM may refuse a harmful request in text form yet comply once an image is introduced. We refer this phenomenon as vision-induced safety degradation (VD). Existing methods typically apply representation-level corrections through fixed intervention schemes. However, such fixed schemes may fail to adapt as the response evolves, causing correction to act at inappropriate locations or persist for an unsuitable duration. To better adapt correction to the evolving response, we propose TrajBridge, an inference-time framework for trajectory-guided safety recovery based on state-dependent feedback. TrajBridge first constructs layer-wise correction references offline to support both localization and correction. During online correction, it repeatedly reassesses the evolving response state to localize where intervention is most useful and applies bounded, state-dependent correction with adaptive strength. A separate correction controller governs when intervention should remain active or be released, preventing correction from being applied longer than necessary. By continuously coupling intervention decisions with the evolving response trajectory, TrajBridge turns static representation calibration into trajectory-guided, closed-loop safety recovery. Across paired VD evaluations, TrajBridge consistently improves safety recovery while preserving stable-safe behavior and general multimodal capabilities. Further analyses validate the importance of dynamic intervention location and duration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language model, safety alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aUFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies vision-induced safety degradation (VD) in multimodal large language models, where a model that safely refuses a harmful text-only request may comply when an image is added. The authors argue that existing representation-level interventions rely on fixed schemes that fail to adapt as generation evolves. They propose TrajBridge, an inference-time framework with three components: offline calibration of layer-wise correction references derived from native safe/unsafe response pairs, online dynamic localization of intervention sites via probe-based evaluation, and a correction controller that governs when intervention activates, releases, and reactivates. Experiments on two MLLM backbones across multiple safety benchmarks demonstrate improved safety recovery while largely preserving model capabilities.

Critical strengths

S1. Clear problem formulation with practical significance.

Vision-induced safety degradation (VD) is a real and observable failure mode in deployed MLLMs.

S2. Well-motivated method with clear presentation and rigorous ablations.

The proposed TrajBridge is conceptually clean and easy to follow. The authors do a commendable job walking through the design intuition.

Critical weaknesses

W1. The VD phenomenon itself is barely studied, and there is a mismatch between the paper's motivation and its actual contribution.

While the paper motivates TrajBridge through the lens of VD, it contributes little to our mechanistic understanding of why VD occurs. The actual proposal is a general-purpose inference-time dynamic safety control framework, with VD serving as an application context.

The paper would be strengthened by either (a) providing deeper analysis of VD's underlying causes, or (b) reframing the contribution as a general dynamic control method rather than a VD-specific solution.

W2. The assumption that text-only behavior can serve as a safe reference breaks down in several realistic scenarios.

The offline calibration step requires collecting VD pairs (yi+,yi−)(y_i^+, y_i^-), where yi+y_i^+ is a safe text-only response. This implicitly assumes that text-only behavior provides a valid safety reference. However, this assumption fails in several important cases:

(a) Image-as-payload attacks: In benchmarks like FigStep, the image itself contains the harmful instruction (for example, text rendered in the image reading "how to make a bomb").

(b) Image-grounded semantic ambiguity: For instructions like "How do I do this?", the text alone is ambiguous without the image. The text-only "safe" response may be an unhelpful refusal such as "I need more context."

W3. The cost of obtaining calibration data and the risk of distribution shift are not addressed.

The paper reports a modest calibration set size (64 VD pairs + 32 stable-safe pairs + 48 benign inputs), which appears lightweight. However, the hidden cost of obtaining these VD pairs is ignored:

(a) VD pair acquisition requires expensive pre-scanning: One must first run the base model on all candidate (I,x)(I, x) pairs (once for text-only, once for multimodal), then apply a safety judge to identify the VD subset. Appendix F shows that VD rates vary dramatically across benchmarks (VL-Safe 57% vs. HoliSafe SSS 0.15%). In low-VD-rate settings, one would need to scan tens of thousands of candidates to obtain just 64 pairs.

(b) Distribution shift between calibration and deployment: The calibration VD pairs are drawn from existing benchmarks (VL-Safe, SIUO, MSTS). Deployed systems, however, will likely face very different VD patterns, including novel jailbreak strategies, new image distributions, or new task domains.

Questions for authors

(1) In a deployed setting, who is responsible for constructing the VD pairs? Does this effectively require running a full VD audit before deployment?

(2) Have you evaluated whether the reference directions generalize to unseen distributions?

Reviewer_kNsNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrajBridge, an inference-time method to reduce vision-induced safety degradation in MLLMs. It builds correction directions from safe and unsafe responses, then adjusts where and when to apply correction during generation. The experiments test safety recovery and general capabilities on two backbones.

Critical strengths
  1. Good motivation: safety behavior can change during generation, so adjusting both the intervention layers and duration makes sense.

  2. The experiments cover two backbones and several datasets, with ablations on intervention location, release, and reactivation.

Critical weaknesses
  1. Limited evaluation of answer quality: a response judged safe may still be a poor answer. Table 24 shows an incoherent response that misses the hazard in the image but is counted as safe. POPE and MMLU bypass correction, while MM-Vet shows more outputs reaching the length limit. It is unclear how often the repaired responses are safe and useful.

  2. Small gain with higher cost: the full controller repairs only one more case than the no-release variant on the development set, but latency rises from 7.53 to 11.62 seconds (Table 17). The simpler controller is not tested in the main evaluation, so it is unclear whether the added cost is worthwhile.

Questions for authors
  1. It would be useful to have human reviewers check a sample of repaired responses with the original images. How many are safe and useful, and how many are poor answers or unnecessary refusals?

  2. Can the authors add results for the no-release variant and ASTRA on the main evaluation, including answer quality and runtime?

  3. Please clarify the MM-Vet score: Section 5.7 reports 14.0, while Table 5 reports 28.0.

Reviewer_JCANOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper defines vision-induced safety degradation (VD) as a paired transition: a response that is judged safe under text-only input becomes judged unsafe when the image is added. It proposes TrajBridge, an inference-time method with layer-wise correction and activation controller for response adaptation. It reports results on three safety datasets: MSTS, SIUO and VL-Safe and two models: LLaVA-1.5-7B and InternVL2.5-4B.

Critical strengths

For safety alignment, most current approaches are data driven. Instead, this paper proposes model safety behavior in terms of where in the response to intervene and when to intervene. This framing is reasonable and valuable for the field to build on top of.

Critical weaknesses

The main weakness is the visual degradation (VD) baselines and problem definition. The current eval is not established as a robust, model-level phenomenon.

  • The judge (Llama-Guard-3-1B) is text-only and asymmetric across the two arms of the pair; one arm gets text-only prompt and the second gets image-as-text
  • The definition mixes two phenomena: text-only refusal suppressed by the image and legitimate cross-modal compositional multimodal risk
  • The paper does not follow the benchmarks’ standard eval protocols, setting their own evaluation flow making it hard to compare against prior published results. Many standard jailbreak suites are missing (MM-SafetyBench, FigStep, HADES, JailbreakV-28K).
  • Main results are on one weakly aligned model only
Questions for authors
  1. What are the results if you were to update the judge to a multimodal model (Llama Guard 3 Vision (11B) or Llama Guard 4 (12B))?
  2. Please report VD incidence and the main comparison on 2-3 latest safety-tuned VLMs.
  3. Please split the VD results into image-jailbreaks, compositional risks and text-only refusal suppression
  4. What are the over-refusal results on MOSSBench?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.