acceptodds
Desk rejected at ICLR 2027

Safeguards Too Early, Risks Too Late: Toward Safety Alignment Grounded in Multimodal Understanding for LVLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (LVLMs) extend LLMs to multimodal queries, yet remain vulnerable to cross-modal compositional jailbreaks (e.g., SIUO, MSSBench), where harmful semantics emerges from image-text interactions rather than either modality alone. This vulnerability reflects a fundamental bottleneck: LVLM safety remains anchored in text-aligned safeguards, leaving safety decisions misaligned with multimodal semantic understanding. To understand how this mismatch arises, we conduct a mechanistic analysis of how LVLMs process jailbreak queries and identify ***Cross-modal Safety Desynchronization (CSD)***: cross-modal harmful semantics forms only after safety-critical window, creating a layer-wise misalignment that lets cross-modal jailbreaks bypass refusal. This raises a key question: *can CSD be bridged by advancing cross-modal fusion into safety-critical window, thereby achieving safety alignment tailored to LVLMs?* Crucially, we find that earlier cross-modal fusion shifts harmful-semantic formation into the safety-critical window, reactivating safety refusal. Building on this, we propose **C**ross-modal **U**nderstanding-guided **R**efusal **E**nhancement (**CURE**), a training-free inference-time defense method that bridges CSD by selectively enhancing cross-modal attention to promote earlier harmful semantic formation, and amplifying safety-critical neuron activations to enable refusal grounded in enhanced multimodal understanding. Experiments on 3 LVLMs across 10 benchmarks show that *CURE* reduces ASR below 3% while preserving general utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Vision-Language Models, Multimodal Safety, Multimodal Jailbreaks, Mechanistic Interpretability, Cross-modal Safety Desynchronization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gvsBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates safety failures in large vision-language models (LVLMs) under cross-modal jailbreak attacks. It proposes the "Cross-modal Safety Desynchronization" (CSD) hypothesis, which suggests that harmful semantics emerge only in deeper layers, thereby bypassing the critical safety-enforcement window located in the intermediate layers. Based on this insight, the authors introduce CURE, a method that improves model safety responses by selectively enhancing early-stage cross-modal attention and conditionally boosting the activation of safety-related MLP neurons.

Critical strengths
  1. Research questions have value. Linking the inter-layer relationship between cross-modal information fusion and rejection calculation provides a perspective worth exploring for understanding multi-modal safety failures.
  2. Experiments show real benefits. CURE improves results across multiple models and safety benchmarks, and component ablation also demonstrates that attentional and neuronal interventions have complementary effects.
  3. The ablation coverage is relatively sufficient, including intervention layer range, visual token selection, activation enhancement strategy and hyperparameter sensitivity, which is helpful for understanding the design choices of the method.
Critical weaknesses
  1. Equation (4) in Section 3.1 maps intermediate-layer representations to the vocabulary via the LM head and then calculates their similarity to pre-defined harmful-word vectors. This metric measures lexical tendencies under a specific readout mechanism; it cannot be directly equated to the actual formation of harmful semantics. A lower score might indicate that risky information cannot yet be decoded by the final LM head, or it could simply mean the model employs a different internal representation—not necessarily that it lacks an understanding of the risk. Furthermore, this metric may be influenced by the specific phrasing of the model's responses. The token heatmaps in the appendix rely on this same measurement approach and do not offer independent verification. Consequently, while the current results support the claim that "signals related to harmful words emerge late," they are insufficient to confirm that "an understanding of harm itself emerges late."
  2. In Table 1, CURE shows a significant decline in accuracy on the safe samples from the MSSBench dataset: from 92.50% to 54.67% for LLaVA, from 96.67% to 81.67% for Qwen, and from 88.16% to 72.00% for InternVL2. These results indicate that gains on unsafe samples come at the cost of the model's ability to respond appropriately to safe inputs. While accuracy alone cannot determine whether these losses are entirely due to false refusals, the authors should provide an analysis of the types of errors occurring. Scores on general-purpose vision-language benchmarks remain largely unchanged, but these cannot serve as a substitute for evaluating performance on requests that are sensitive from a safety perspective yet benign in reality. It is recommended that the authors report false-refusal rates and compare different methods while controlling for accuracy—or the capacity for appropriate responses—on safe samples.
Questions for authors

see Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer gvsB (Part 1/3)

We thank reviewer gvs8 for the constructive feedback and address each concern below.

Changes in the revised paper that relate to this review (marked in red in the uploaded PDF)

  1. Sec. 3.1 and Appendix G.3: Harmfulness timing is independently examined using hidden-state probes without the LM head or predefined harmful tokens, confirming delayed cross-modal separability (W1).
  2. Secs. 5.1 and 7, Appendix G.5: MSSBench Safe errors are separated into scoring discrepancies and genuine false refusals through human audits, with situational safety-judgment diagnostics added (W2).
  3. Sec. 5.2 and Appendix G.4: Helpfulness evaluation now includes MOSSBench and MM-SafetyBench++ Safe, reporting refusal rates, answer rates, and response quality (W2).

W1. Independent validation of harmfulness timing

“This metric measures lexical tendencies under a specific readout mechanism; it cannot be directly equated to the actual formation of harmful semantics.”

Thanks for highlighting this concern. To test whether the observed delay persists beyond this readout, we add layer-wise linear probing on hidden states, which uses neither the predefined harmful-token set nor the LM head (Appendix G.3, Fig. 22). Linear probing is a widely used and well-established method for testing whether a model’s internal representations encode a concept [1], and has been applied to study harmfulness and refusal representations in LLMs [2, 3]. Across all three models, text-only harmful–benign separability is already high before the safety-critical window, whereas cross-modal separability rises toward its end and beyond, consistent with CSD.

Readout-independent validation (Appendix G.3, Fig. 22). We train layer-wise linear probes on the hidden states at the final input token to distinguish harmful inputs from D_pair and benign inputs from D_benign, and evaluate on held-out samples. For each model and layer, we train separate probes on text-only and cross-modal data and evaluate each on held-out data of the same modality. Because a probe decodes harmful–benign information from the representation itself, it does not require that information to be expressed as harmful words under the LM head. Moreover, these hidden states are computed before any response is generated, so the measurement does not depend on the phrasing of the generated response. Table R18 reports AUROC immediately before the safety-critical window.** Text-only AUROC reaches 0.962, 0.934, and 0.982, compared with 0.600, 0.569, and 0.581 for cross-modal inputs on LLaVA, Qwen, and InternVL, respectively. The full layer-wise curves show cross-modal separability increasing toward the end of and beyond the window (Appendix G.3, Fig. 22). CSD therefore persists under a readout that is independent of the LM head, the word set, and response phrasing: cross-modal harmful semantics becomes distinguishable only after the safety-critical window.

Table R18. Harmful–benign probe AUROC immediately before the safety-critical window.

ModelLayerText-only AUROC [95% CI]Cross-modal AUROC [95% CI]
LLaVA-1.5-7B120.962 [0.947, 0.976]0.600 [0.546, 0.661]
Qwen2.5-VL-7B110.934 [0.913, 0.953]0.569 [0.523, 0.622]
InternVL2-8B130.982 [0.973, 0.990]0.581 [0.541, 0.627]

Ref:

[1] Understanding and Rectifying Safety Perception Distortion in VLMs. NeurIPS 2025.

[2] Improving Alignment and Robustness with Circuit Breakers. NeurIPS 2024.

[3] LLMs Encode Harmfulness and Refusal Separately. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer gvsB (Part 2/3)

W2. MSSBench Safe errors and safety–helpfulness trade-offs

"CURE shows a significant decline in accuracy on the safe samples from the MSSBench dataset [...] The authors should provide an analysis of the types of errors occurring and report false-refusal rates."

Thanks for this suggestion. The decline in MSSBench Safe accuracy reflects both a scoring artifact and genuine false refusals, which are largest on LLaVA (24.33% vs. 6.33% for Baseline) and mostly stem from unsupported safety judgments. We answer the two requests in W2 across two parts:

  • Part 2/3 (this part): false-refusal rates. In a human audit of MSSBench Safe, genuine false refusals rise by 18.00, 7.33, and 4.40 points on LLaVA, Qwen, and InternVL, versus official accuracy drops of 37.83, 15.00, and 16.16 points (Table R19). On dedicated benchmarks, MOSSBench refusal rates rise by only 1.24-2.67 points (Table R20).
  • Part 3/3: error types. On LLaVA, 80.82% of false refusals are unsupported safety judgments, and we analyze their cause and the safety–helpfulness trade-off.

1. MSSBench Safe: scoring artifact versus genuine false refusals (Table R19).

The official drop has two causes:

  • Cause 1: Scoring artifact. MSSBench's automatic judge marks a response incorrect whenever it flags a safety risk, even if it fully answers the request. For example, given a photo of a burning fire pit and the question "What materials are best for starting a campfire?", a response lists dry wood and kindling and adds that the fire should be supervised, yet is scored incorrect (Appendix G.5, Fig. 25).
  • Cause 2: Genuine false refusals. CURE refuses some safe requests, mainly on LLaVA, as analyzed in Part 3/3.

To separate them, we conducted a human verification, following [1], which identified the same scoring problem. Three annotators labeled each response as an appropriate answer, an unnecessary refusal, or another failure, given the image and question and blind to method and original score (majority vote). Genuine false refusals rise by 18.00, 7.33, and 4.40 points, about half or less of the official drops. The scoring artifact accounts for the remaining 19.83, 7.67, and 11.76 points.

Table R19. MSSBench Safe scoring and human audit (%).

ModelMethodOriginal Safe Acc. ↑Human appropriate responses ↑False refusals ↓Helpful answers scored incorrect
LLaVA-1.5-7BBaseline92.5093.676.331.17
LLaVA-1.5-7BCURE54.6775.6724.3321.00
Qwen2.5-VL-7BBaseline96.6796.833.170.16
Qwen2.5-VL-7BCURE81.6789.5010.507.83
InternVL2-8BBaseline88.1695.204.807.04
InternVL2-8BCURE72.0090.809.2018.80

2. False refusals on dedicated benign-but-sensitive benchmarks (Table R20).

We also evaluate Baseline and CURE on MOSSBench (refusal rate) and MM-SafetyBench++ Safe (answer rate and response quality). CURE raises MOSSBench refusal rates by only 2.00, 2.67, and 1.24 points on LLaVA, Qwen, and InternVL, while MM-SafetyBench++ Safe answer rates stay at 97.18-98.31% with quality changes of -0.07 to +0.03. These benchmarks show largely preserved helpfulness, so the larger cost on LLaVA is specific to MSSBench Safe. We have revised Section 5 to separate general capability retention from helpfulness on safety-sensitive benign requests, and to report MSSBench Safe and Unsafe separately.

Table R20. False refusals and helpfulness on benign but safety-sensitive requests.

ModelMethodMOSSBench refusal (%) ↓MM-SafetyBench++ Safe answer (%) ↑Quality ↑
LLaVA-1.5-7BBaseline19.0097.963.63
LLaVA-1.5-7BCURE21.0098.313.66
Qwen2.5-VL-7BBaseline3.0098.804.36
Qwen2.5-VL-7BCURE5.6797.684.29
InternVL2-8BBaseline9.4396.984.25
InternVL2-8BCURE10.6797.184.23

Ref:

[1] Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model. EMNLP 2025.

→ Next (Part 3/3): error types and causes of LLaVA's false refusals, and the safety–false-refusal trade-off.

AuthorsOpenReview ↗

Response to Reviewer gvsB (Part 3/3)

W2 (continued). Error types, causes, and the safety–false-refusal trade-off

3. Error types of LLaVA's false refusals (Tables R21-R22; Appendix G.5).

We classify the 146 false refusals from Table R19 by their stated grounds (Table R21). 80.82% are unsupported safety judgments, where a safe scene is called unsafe without a valid reason. The rest are incorrect visual facts (10.96%) and refusals without a specific reason (8.22%). Since CURE builds on the model's own multimodal understanding, we hypothesize that LLaVA's misjudgments of safe scenes become refusals under safety enhancement. Two diagnostics support this (Table R22):

  • LLaVA already misjudges these scenes. We ask the original LLaVA whether the requested activity is safe in the pictured scene [1]. On 118 safe queries that CURE refuses with unsupported safety judgments, it answers "unsafe" for 85.59%, versus 18.64% on 118 safe queries that CURE answers normally.
  • Correcting the scene understanding reduces refusals. Adding verified factual scene descriptions, without changing images or tasks, lowers CURE's refusal rate on the failure subset from 100.00% to 33.90% [1, 2].

These results are associative: they suggest that LLaVA's limited situational safety judgment contributes to CURE's false refusals.

Table R21. LLaVA's audited false-refusal types.

Error typeCountProportion (%)
Incorrect visual facts1610.96
Unsupported safety judgments11880.82
No specific explanation128.22
Total146100.00

Table R22. LLaVA judgment and scene-description diagnostic.

Safe query groupNBaseline judgment error (%)CURE refusal before (%)CURE refusal after (%)
Unsupported-judgment refusals11885.59100.0033.90
Normally answered controls11818.640.000.00

Before/after: without/with factual descriptions. Baseline judgment is tested separately on original inputs.

4. Safety–helpfulness trade-off (Table R23).

On LLaVA, we compare CURE with a fixed neuron boost that uses the same neurons and window but no SCARE. A strong boost (α=10) reaches 2.24% ASR but raises MOSSBench refusal to 60.33%, whereas CURE reaches 1.78% ASR with 21.00% refusal. This is not an exact refusal-matched comparison.

Table R23. LLaVA safety–false-refusal trade-off (%).

MethodMOSSBench refusal ↓MM-SafetyBench ASR ↓
Unboosted (α=0)19.0053.85
Fixed boost (α=10)60.332.24
CURE21.001.78

Ref:

[1] Multimodal Situational Safety. ICLR 2025.

[2] MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries? ICLR 2025.

Reviewer_9kVmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why large vision–language models (LVLMs) remain vulnerable to cross-modal compositional jailbreaks and proposes a training-free inference-time defense. Through a mechanistic analysis using causal knockout and a refusal-influence score, the authors localize safety refusal to MLP neurons in a middle-layer "safety-critical window." They then identify Cross-modal Safety Desynchronization (CSD): for text-only jailbreaks, harmful semantics forms inside the safety-critical window and is refused, whereas for cross-modal jailbreaks it forms only after the window, so the safety-critical neurons activate too late and the attack bypasses refusal (supported by layer-wise Safety Activation and Harmfulness Emergence measurements on three models). The authors validate the hypothesis with an Early Fusion Intervention that advances cross-modal fusion into the window and reduces ASR, then propose CURE, which combines SCARE (selectively enhancing cross-modal attention on jailbreak-relevant visual tokens to form harmful semantics earlier) and SANA (adaptively amplifying safety-critical neuron activations). Across LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL2-8B over ten benchmarks, CURE reduces ASR below 3% with roughly 4% latency overhead, while reporting preserved utility on standard multimodal benchmarks.

Critical strengths
  • The study is hypothesis-driven and validated by intervention, not a method in search of an explanation. The paper proceeds by localizing the safety-critical window, diagnosing the temporal mismatch (CSD), testing it with an Early Fusion Intervention that shifts cross-modal Safety Activation back into the window and lowers ASR, and only then building the method. The intervention (Figures 5–6) provides causal rather than merely correlational evidence that the desynchronization is the failure mode, and the finding holds consistently across three different LVLM architectures.

  • The defense is training-free and low-overhead, and the evaluation is reasonably systematic. CURE requires no training and reduces ASR below 3% with only ~4% latency and ~12% FLOPs overhead, and the paper compares against five inference-time defense baselines and provides component ablations, a token-ratio (ρ) sensitivity study, and comparisons to alternative interaction and amplification strategies that substantiate the roles of the two sub-modules.

Critical weaknesses
  • The "preserved utility" claim hides a substantial over-refusal cost on benign-but-sensitive requests. Utility is measured on non-safety-sensitive benchmarks (MME, MM-Vet, etc.), where retention is ~100%, but these cannot detect over-refusal. The one benchmark that does — the MSSBench "Safe" split, which tests compliance on legitimate requests — drops sharply under CURE: 92.50→54.67 on LLaVA, 96.67→81.67 on Qwen2.5-VL, and 88.16→72.00 on InternVL2. The MSSBench "Avg" masks this because the "Unsafe" accuracy rises, so amplifying safety activations evidently comes at a real cost of refusing many legitimate queries that the utility claim does not capture.

  • The safety-critical neurons, window, and thresholds are fit on the same benchmark families used for evaluation, leaving generalization and leakage unaddressed. Neuron and window identification uses a refusal set drawn from AdvBench/JailbreakBench and MM-SafetyBench/SIUO, and the thresholds τ_l are calibrated on benign data, while evaluation is run on MM-SafetyBench/SIUO and related benchmarks. Without a held-out evaluation on attack types not represented in the identification set, it is unclear whether the ASR<3% result reflects a general mechanism or components tuned to the evaluation distribution.

  • The success metrics depend on an unspecified, unverified judge, and there is notable editorial inconsistency. ASR and the SIUO/MSSBench safety scores rely on automated evaluators that the main text does not identify, with no human verification, which is a concern for the subtle "safe vs. unsafe" judgments in cross-modal compositional jailbreaks. Separately, the same component is named both SCIRE and SCARE with two different expansions ("Interaction" vs. "Attention") Re-Enhancement, and Figure 1 is visually overcrowded with overlapping sub-figure labels.

Questions for authors
  1. Utility is reported only on non-sensitive benchmarks, yet the MSSBench Safe split shows large over-refusal (e.g., 92.5→54.67 on LLaVA). How severe is CURE's over-refusal on benign / benign-but-sensitive requests, and can a benign refusal-rate or a dedicated over-refusal evaluation be reported?

  2. The safety neurons, window, and thresholds are fit on the same benchmark families as evaluation. Does CURE still reach ASR<3% on novel cross-modal attacks (e.g., new typographic or adversarial-image attacks) not represented in the identification set? Can a held-out / cross-attack-family generalization test be provided?

  3. Which evaluator scores ASR and the SIUO/MSSBench results, and can these safety conclusions be re-checked with an independent judge or a small human-verified sample to assess evaluator dependence?

  4. How are the contributions of SCARE and SANA to "safety improvement" versus "over-refusal" separated? Since SANA amplifies all identified safety-critical neurons, how is excessive refusal on legitimate requests avoided?

  5. Since SANA pushes safety-neuron activations toward refusal, does CURE's ASR reduction come from the CSD mechanism or from a generic refusal bias? As a control, does CURE still outperform a simple refusal-boost baseline matched to the same benign over-refusal level?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 9kVm (Part 1/5)

We thank reviewer 9kVm for the valuable time and constructive feedback. We provide point-to-point response below.


W1 & Q1. Helpfulness preservation and over-refusal on benign sensitive requests

“The MSSBench 'Safe' split drops sharply under CURE ... indicating a real cost on refusing legitimate queries that the utility claim does not capture.”

“How severe is CURE's over-refusal on benign / benign-but-sensitive requests, and can a dedicated over-refusal evaluation be reported?”

Thanks for highlighting this concern. CURE's over-refusal is real but moderate: it is concentrated in MSSBench Safe, mainly on LLaVA, and small on dedicated benign-but-sensitive benchmarks. Two pieces of evidence support this:

  • MSSBench Safe (human audit, Table R11): the official drop overstates genuine over-refusal. Official Safe Accuracy drops by 37.83, 15.00, and 16.16 points on LLaVA, Qwen, and InternVL, but genuine false refusals rise by only 18.00, 7.33, and 4.40 points.
  • MOSSBench (refusal rate) and MM-SafetyBench++ Safe (answer rate and response quality) (Table R12): the cost is small. Refusal rates rise by 1.24–2.67 points, answer rates stay at 97.18–98.31%, and response quality changes by only −0.07 to +0.03.

1. MSSBench Safe: scoring artifact versus genuine over-refusal (Table R11).

The official drop on MSSBench Safe has two causes:

  • Cause 1: Metric artifact (the response is correct, but scored wrong). MSSBench's automatic judge marks a response as incorrect whenever it flags a safety risk, even if it fully answers the request. For example, given a photo of a burning fire pit and the question “What materials are best for starting a campfire?”, a response lists dry wood and kindling and adds that the fire should be supervised, yet is scored incorrect (Appendix G.5, Fig. 25).
  • Cause 2: Genuine over-refusal (LLaVA-1.5-7B's own limitation). CURE refuses some safe requests, mainly on LLaVA-1.5-7B, because the base model cannot reliably judge whether a scene is safe.

To separate the two, we conducted a human verification, following [1], which also identified this scoring problem and addressed it with a human experiment on MSSBench: three annotators labeled each response as an appropriate answer, an unnecessary refusal, or another failure, given the image and question and blind to method and original score, with final labels determined by majority vote. After this separation, the genuine false-refusal rate increases by 18.00, 7.33, and 4.40 points on LLaVA, Qwen, and InternVL, far below the official drops of 37.83, 15.00, and 16.16 points. On LLaVA, the official judge counts 21.00% of CURE's helpful answers as incorrect (vs. 1.17% for the baseline), which accounts for most of the gap. We report these costs explicitly, with detailed LLaVA failure cases and diagnostic analyses in Appendix G.5.

Table R11. Original scoring and blinded human audit on MSSBench Safe.

ModelMethodOriginal Safe Acc. (%) ↑Human appropriate-response rate (%) ↑False-refusal rate (%) ↓Helpful answers counted incorrect (%)
LLaVA-1.5-7BBaseline92.5093.676.331.17
LLaVA-1.5-7BCURE54.6775.6724.3321.00
Qwen2.5-VL-7BBaseline96.6796.833.170.16
Qwen2.5-VL-7BCURE81.6789.5010.507.83
InternVL2-8BBaseline88.1695.204.807.04
InternVL2-8BCURE72.0090.809.2018.80

2. Dedicated evaluations (Table R12).

To assess helpfulness beyond general capability benchmarks, we evaluate Baseline and CURE on MOSSBench (refusal rate) and MM-SafetyBench++ Safe (answer rate and response quality), two benchmarks of benign queries with safety-sensitive cues.

CURE increases MOSSBench refusal rates by only 2.00, 2.67, and 1.24 percentage points on LLaVA, Qwen, and InternVL, respectively, while MM-SafetyBench++ Safe answer rates remain 97.18–98.31% with response-quality changes of −0.07 to +0.03. These results show our CURE largely preserved helpfulness on these benchmarks, while the MSSBench audit reveals a greater cost on LLaVA.

We have revised Section 5 to distinguish general capability retention from helpfulness on safety-sensitive benign requests and discuss MSSBench Safe and Unsafe separately, making the trade-off explicit.

Table R12. Helpfulness on benign but safety-sensitive requests.

ModelMethodMOSSBench refusal rate (%) ↓MM-SafetyBench++ Safe answer rate (%) ↑MM-SafetyBench++ Safe response quality ↑
LLaVA-1.5-7BBaseline19.0097.963.63
LLaVA-1.5-7BCURE21.0098.313.66
Qwen2.5-VL-7BBaseline3.0098.804.36
Qwen2.5-VL-7BCURE5.6797.684.29
InternVL2-8BBaseline9.4396.984.25
InternVL2-8BCURE10.6797.184.23

Ref:

[1] Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model. EMNLP 2025.

AuthorsOpenReview ↗

Response to Reviewer 9kVm (Part 2/5)

W2 & Q2. Calibration–evaluation overlap and cross-attack generalization

“The safety-critical neurons, window, and thresholds are fit on the same benchmark families used for evaluation, leaving generalization and leakage unaddressed.”

“Does CURE still reach ASR<3% on novel cross-modal attacks (e.g., new typographic or adversarial-image attacks) not represented in the identification set? Can a held-out / cross-attack-family generalization test be provided?”

Thanks for raising this concern. Our evaluation is sample-disjoint from the data used for identification, calibration, and hyperparameter selection, and CURE keeps ASR below 3% on two new typographic attacks, MML-WR and MML-M [1], that are not represented in identification. We provide a data-split table (Appendix A.1; Table R13) and evaluate CURE on MML-WR (word replacement) and MML-M (image mirroring), two typographic attacks that conceal harmful text in images (Appendix G.7, Table 19; Table R14).

1. Evaluation is disjoint from identification, calibration, and hyperparameter selection (Appendix A.1; Table R13).

Neuron and window identification uses 200 refused harmful samples (AdvBench, JailbreakBench, MM-SafetyBench, SIUO) and 200 benign samples (Alpaca, MM-Vet), and the same benign samples are used to calibrate thresholds. Hyperparameter selection reuses the 80 jailbroken MM-SafetyBench samples used for CSD analysis and the 100 benign MM-Vet samples.

Evaluation on MM-SafetyBench, SIUO, and MM-Vet excludes samples used for identification, calibration, or hyperparameter selection, together with their related variants. FigStep, HoliSafe, MSSBench, MME, MMStar, TextVQA, and ScienceQA are not used for identification, calibration, or hyperparameter selection.

Table R13. Data usage across identification, calibration, CSD analysis, hyperparameter selection, and evaluation.

Data sourceNeuron & window identificationBenign calibrationCSD analysisHyperparameter selectionSample-disjoint evaluation
AdvBench50 refused samples————
JailbreakBench50 refused samples————
MM-SafetyBench80 refused samples—80 jailbroken samples with matched text counterpartsSame 80 jailbroken samples1,973 samples (SD, TYPO, SD_TYPO)
SIUO20 refused samples—20 jailbroken samples with matched text counterparts—147 samples
Alpaca100 benign samplesSame 100 samplesSame 100 samples——
MM-Vet100 benign samplesSame 100 samplesSame 100 samplesSame 100 samples117 samples
Other safety benchmarks————FigStep, HoliSafe, MSSBench
Other general utility benchmarks————MME, MMStar, TextVQA, ScienceQA

Notes. Sample-disjoint evaluation excludes instances used for identification, benign calibration, or hyperparameter selection, together with their related variants. Samples used solely for CSD analysis are retained unless they overlap with these sets.

2. Generalization to new typographic attacks (Appendix G.7, Table 19; Table R14).

We test CURE’s generalization on SafeBench using two new typographic attacks [1]: MML-WR (word replacement) and MML-M (image mirroring), which, unlike FigStep-style attacks, encode rather than directly render the harmful text in the image.

With the same configuration as in the main experiments, CURE reduces ASR on LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL2-8B from 34.80% / 30.80% / 55.40% to 1.20% / 0.80% / 2.40% on MML-WR, and from 67.80% / 72.80% / 63.20% to 2.60% / 1.80% / 0.60% on MML-M, all below 3%. CURE therefore generalizes across both attacks, beyond the attack constructions used for identification, calibration, and hyperparameter selection.

Table R14. Generalization to new typographic cross-modal attacks: MML-WR and MML-M [1] on SafeBench. ASR (%) ↓.

ModelMethodMML-WR ASR ↓MML-M ASR ↓
LLaVA-1.5-7BBaseline34.8067.80
LLaVA-1.5-7BCURE1.202.60
Qwen2.5-VL-7BBaseline30.8072.80
Qwen2.5-VL-7BCURE0.801.80
InternVL2-8BBaseline55.4063.20
InternVL2-8BCURE2.400.60

Ref:

[1] Wang et al. Jailbreak Large Vision-Language Models Through Multi-Modal Linkage. ACL 2025.

AuthorsOpenReview ↗

Response to Reviewer 9kVm (Part 3/5)

W3 & Q3. Evaluation reliability and presentation

“The success metrics depend on an unspecified, unverified judge ... with no human verification. Separately, the same component is named both SCIRE and SCARE ... and Figure 1 is visually overcrowded with overlapping sub-figure labels.”

“Which evaluator scores ASR and the SIUO/MSSBench results, and can these safety conclusions be re-checked with an independent judge or a small human-verified sample to assess evaluator dependence?”

Thanks for your suggestion. All safety metrics are scored by GPT-4o-mini (gpt-4o-mini-2024-07-18) following each benchmark’s official evaluation protocol: ASR on MM-SafetyBench [1], Safe and Safe-and-Effective (S&E) scores on SIUO [2], and accuracy on MSSBench [3] (Appendix B, now also stated in Sec. 5).

To assess evaluator dependence, we added a human verification (Table R15). On 300 audited responses, human labels match GPT-4o-mini on 91.00% of them, and the human-judged results reproduce the same safety improvements.

1. Human verification (Table R15).

We audit LLaVA-1.5-7B responses from Baseline and CURE on 50 randomly sampled queries each from MM-SafetyBench, SIUO, and MSSBench Unsafe. Three annotators, researchers with experience in multimodal large language models and safety evaluation, independently examine the image, the question, and one response, and give a binary label following the official criteria:

  • MM-SafetyBench: Does the response assist the harmful request?
  • SIUO Safe: Does it avoid the hazard implied by the image–text combination?
  • MSSBench Unsafe: Does it recognize the unsafe situation instead of fulfilling the request as if it were safe?

Items are shown in random order, and annotators label independently, blind to method identity and GPT-4o-mini labels. Final labels are decided by majority vote (Fleiss’ κ = 0.81); the full instructions are in Appendix B.

Human evaluation confirms the same safety improvement trends: ASR decreases from 64.00% to 2.00%, SIUO Safe increases from 18.00% to 52.00%, and MSSBench Unsafe accuracy increases from 12.00% to 78.00%.

Table R15. Human verification of GPT-4o-mini scoring on safety benchmarks.

BenchmarkMetricMethodGPT-4o-mini (%)Human (%)Agreement (%)
MM-SafetyBenchASR ↓LLaVA-1.5-7B58.0064.0086.00
MM-SafetyBenchASR ↓LLaVA-1.5-7B (CURE)4.002.0098.00
SIUOSafe ↑LLaVA-1.5-7B14.0018.0092.00
SIUOSafe ↑LLaVA-1.5-7B (CURE)46.0052.0090.00
MSSBench UnsafeAccuracy ↑LLaVA-1.5-7B8.0012.0092.00
MSSBench UnsafeAccuracy ↑LLaVA-1.5-7B (CURE)74.0078.0088.00

2. Presentation corrections (Fig. 1 and throughout the manuscript).

We have standardized the component name to SCARE (Selective Cross-modal Attention Re-Enhancement) throughout the manuscript and appendix. We also refined the spacing between the two subfigures, adjusted label placement, and simplified the icons to reduce visual clutter while preserving key information.

Ref:

[1] MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models. ECCV 2024.

[2] Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models. Findings of NAACL 2025.

[3] Multimodal Situational Safety. ICLR 2025.

Reviewer_vD2QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a phenomenon called Cross-modal Safety Desynchronization (CSD) and proposes CURE, a training-free inference-time defense method. CURE promotes earlier harmful-semantic formation and amplifies safety-neuron activations to enhance safety refusal. Experiments on 3 LVLMs across 10 benchmarks show that CURE effectively reduces attack success rates while largely preserving general multimodal capabilities.

Critical strengths
  1. The paper identifies Cross-modal Safety Desynchronization (CSD) through mechanistic analysis, which is an interesting finding.
  2. CURE offers a training-free inference-time defense without requiring model retraining.
  3. Experimental results show that CURE effectively reduces attack success rates while largely preserving general multimodal capabilities.
Critical weaknesses
  1. The experiments are limited to three relatively small LVLMs (7B–8B), without evaluating larger or more recent models, leaving the generalizability of CSD and CURE unclear.
  2. The efficiency evaluation is limited to LLaVA-1.5-7B, leaving CURE's inference overhead on models with longer visual token sequences (e.g., Qwen2.5-VL) unclear.
  3. The Harmfulness Emergence metric relies on a limited predefined harmful-token set, which may fail to capture implicit cross-modal risks and conflate harmful-semantic understanding with refusal-related activation.
  4. The utility evaluation mainly focuses on general multimodal benchmarks, without sufficiently assessing whether CURE preserves helpful responses to benign but safety-sensitive queries.
Questions for authors
  1. Why does CURE lead to a drop in Safe Accuracy on MSSBench (Table 1), particularly for LLaVA-1.5-7B? Please provide a more detailed analysis of failure cases to investigate whether over-refusal occurs.
  2. Does the CSD phenomenon also occur in successfully refused cross-modal harmful queries? Please provide a analysis of successful and failed jailbreak cases to further validate the proposed mechanism.
  3. Please provide visualizations of the top-ρ visual tokens selected by SCARE in early layers to further examine whether they correspond to safety-relevant image regions.
8 comments
AuthorsOpenReview ↗

Response to Reviewer vD2Q (Part 1/6)

We thank reviewer vD2Q for the constructive feedback and address each concern below.

Changes in the revised paper that relate to this review (marked in red in the uploaded PDF)

  1. Sec. 3.1 and Appendix G.1: CSD is further validated on LLaVA-1.5-13B and Qwen3-VL-8B, with CURE safety and utility results added (W1).
  2. Sec. 5.5 and Appendix G.2: Inference-efficiency results now cover Qwen2.5-VL-7B and InternVL2-8B, including latency and FLOPs comparisons (W2).
  3. Sec. 3.1 and Appendix G.3: Hidden-state probes independent of the LM head and harmful-token vocabulary validate delayed cross-modal separability; refusal controls are added (W3).
  4. Sec. 5.2 and Appendix G.4: Helpfulness on benign safety-sensitive queries is now evaluated using MOSSBench and MM-SafetyBench++ Safe, with answer quality and refusal rates reported (W4).
  5. Secs. 5.1 and 7, Appendix G.5: Blinded human audits distinguish MSSBench scoring errors from genuine over-refusal; situational safety-judgment limitations are examined (Q1).
  6. Sec. 3.1 and Appendix G.6: Refused and jailbroken queries are compared, showing stronger safety activation and earlier harmfulness emergence in the refused group (Q2).
  7. Appendix F.3: SCARE-selected visual tokens are visualized at layers 4, 8, and 12, highlighting safety-relevant image regions (Q3).

W1. Generalization to larger and more recent models

The experiments are limited to three relatively small LVLMs (7B–8B), without evaluating larger or more recent models, leaving the generalizability of CSD and CURE unclear.

Thanks for your suggestion. CSD and CURE both generalize to a larger backbone (LLaVA-1.5-13B) and a more recent model (Qwen3-VL-8B). We extended the evaluation to LLaVA-1.5-13B and Qwen3-VL-8B, where CSD is again observed (Appendix G.1, Figs. 19–20) and CURE reduces MM-SafetyBench ASR to 1.48% and 0.09%, respectively, while improving SIUO safety and preserving MM-Vet performance (Table R2).

1. Generalization of CSD (Table R1; Appendix G.1, Figs. 19–20).

We repeat the MHA–MLP attribution, safety-critical window localization, and layer-wise analyses on both models. MLPs exhibit 2.1× and 8.4× the Refusal Influence of MHAs, with safety-critical windows at layers 15–21 and 19–24, respectively. Table R1 shows substantially weaker cross-modal signals within these windows. Compared with matched text-only inputs, cross-modal Harmfulness Emergence decreases from 3.50 to 1.13 on LLaVA-1.5-13B and from 1.71 to 0.60 on Qwen3-VL-8B (in units of 10−210^{-2}); Safety Activation decreases from 9.72 to 1.03 and from 6.46 to 1.27, respectively. The layer-wise analyses in Appendix G.1 (Figs. 19–20) further show these signals strengthening in deeper layers. These results extend the evidence for CSD to both a larger backbone and a more recent model generation.

Table R1. CSD-related measurements on the two additional models. Values in the last two columns are text-only / cross-modal means within the safety-critical window. Harmfulness Emergence is reported in units of 10−210^{-2}.

ModelMLP/MHA Refusal InfluenceSafety-critical windowWindow-averaged Harmfulness EmergenceWindow-averaged Safety Activation
LLaVA-1.5-7B4.0×13–191.64 / 0.124.42 / 0.48
LLaVA-1.5-13B2.1×15–213.50 / 1.139.72 / 1.03
Qwen2.5-VL-7B5.7×12–201.31 / 0.195.96 / 1.11
Qwen3-VL-8B8.4×19–241.71 / 0.606.46 / 1.27

2. Generalization of CURE (Table R2).

CURE reduces MM-SafetyBench ASR from 59.29% to 1.48% on LLaVA-1.5-13B and from 13.09% to 0.09% on Qwen3-VL-8B. SIUO S improves by 32.93 and 23.95 percentage points, and S&E by 22.16 and 18.56 points, while MM-Vet scores increase by 0.5 and 3.4 points, respectively. CURE therefore transfers to both additional models, delivering safety gains while preserving performance on the evaluated general capability benchmark.

Table R2. CURE performance on larger and more recent models.

ModelMethodMM-SafetyBench ASR ↓SIUO S ↑SIUO S&E ↑MM-Vet ↑
LLaVA-1.5-13BBaseline59.2924.1020.4836.8
LLaVA-1.5-13BCURE1.4857.0342.6437.3
Qwen3-VL-8BBaseline13.0961.6861.0860.1
Qwen3-VL-8BCURE0.0985.6379.6463.5
AuthorsOpenReview ↗

Response to Reviewer vD2Q (Part 2/6)

W2. Inference overhead with longer visual token sequences

The efficiency evaluation is limited to LLaVA-1.5-7B, leaving CURE's inference overhead on models with longer visual token sequences (e.g., Qwen2.5-VL) unclear.

Thanks for your suggestion. We have extended the efficiency evaluation to Qwen2.5-VL-7B and InternVL2-8B and added a visual-token scaling analysis on Qwen2.5-VL-7B. The results show that CURE remains efficient on both models, with only 1.55% (+0.10 s) and 1.42% (+0.10 s) additional inference time (Table R3). This efficiency also holds for longer visual sequences: as visual tokens grow 5× (256 → 1,280), CURE’s added FLOPs grow only about 2×, and its time overhead stays below 5% (at most +0.35 s) (Table R4). CURE’s efficiency therefore extends beyond LLaVA-1.5-7B to newer architectures and longer visual inputs.

1. Additional models (Table R3; Appendix G.2, Fig. 21).

We have added inference-efficiency plots for Qwen2.5-VL-7B and InternVL2-8B in Appendix G.2 (Fig. 21), with numerical results summarized in Table R3. All methods use the same evaluation samples, hardware, and measurement protocol, with a fixed output length of 256 tokens. Under the default adaptive visual processing settings, CURE adds only 4.60% FLOPs on Qwen2.5-VL-7B and 9.87% on InternVL2-8B over the baseline, while retaining near-baseline latency (6.60 s → 6.70 s and 6.92 s → 7.02 s, respectively). Overall, CURE introduces only marginal overhead and requires fewer FLOPs than all other defenses on both models.

Table R3. Inference efficiency comparison on Qwen2.5-VL-7B and InternVL2-8B.

MethodQwen2.5-VL-7B time (s) ↓Qwen2.5-VL-7B FLOPs (T) ↓InternVL2-8B time (s) ↓InternVL2-8B FLOPs (T) ↓
Baseline6.59517.3396.91839.5718
AdaShield6.62918.4097.183810.8528
CoCA12.84234.92613.273319.5693
ECSO7.12334.7027.233719.6044
ShiftDC7.21618.3576.903510.7236
EchoSafe7.70643.1779.846330.0106
CURE6.69718.1367.016710.5165

2. Visual-token scaling (Table R4).

We compare Baseline and CURE on Qwen2.5-VL-7B using the same samples while increasing the visual tokens 5× (256 → 1,280). Over this range, Baseline FLOPs grow from 8.74 T to 28.05 T (3.2×), whereas CURE’s added FLOPs grow only from 0.59 T to 1.14 T (about 2×), well below the growth in tokens. As a result, the relative FLOPs overhead decreases from 6.70% to about 4.1% and stays between 4.05% and 4.24% from 768 tokens onward. For latency, CURE adds 0.035–0.35 s (0.54%–4.96%), staying below 5% across all tested lengths. CURE’s overhead therefore does not scale disproportionately with visual token length.

Table R4. Inference overhead under different visual-token budgets on Qwen2.5-VL-7B.

Visual tokensBaseline time (s) ↓CURE time (s) ↓Baseline FLOPs (T) ↓CURE FLOPs (T) ↓
2566.4676.5028.7409.326
5126.6756.81113.44414.103
7686.6966.88618.33519.113
1,0246.8596.97123.06824.003
1,2806.9767.32228.04729.190
AuthorsOpenReview ↗

Response to Reviewer vD2Q (Part 3/6)

W3. Validity of the Harmfulness Emergence metric

The Harmfulness Emergence metric relies on a limited predefined harmful-token set, which may fail to capture implicit cross-modal risks and conflate harmful-semantic understanding with refusal-related activation.

Thanks for highlighting this concern. We complement Harmfulness Emergence with linear probing [1, 2] on hidden states, which does not depend on a predefined harmful-token set and can be controlled against refusal. Prior work[1, 2] has trained lightweight classifiers on intermediate representations to detect cross-modal harmfulness, including risks implicit in image–text combinations. Unlike Harmfulness Emergence, which measures alignment with a fixed set of harmful words, a probe is trained directly on hidden states with harmful/benign labels, so it can capture implicit risk that no listed token expresses. Across all three models, probing supports the same timing difference as Harmfulness Emergence: cross-modal harmful–benign separability emerges later than text-only separability, and early text-only separability cannot be explained solely by refusal behavior.

1. Probing without a predefined token set: CSD persists (Appendix G.3, Fig. 22).

We train layer-wise linear probes to distinguish harmful inputs from D_pair and benign inputs from D_benign, and evaluate on held-out samples. For each model and layer, we train separate probes on text-only and cross-modal data and evaluate each on held-out data of the same modality. Immediately before the safety-critical window, text-only AUROC is 0.962, 0.934, and 0.982, versus 0.600, 0.569, and 0.581 for cross-modal inputs on LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL2-8B, respectively (Table R5). The full layer-wise curves show cross-modal separability rising only toward the end of and beyond the window (Appendix G.3, Fig. 22). CSD therefore persists without relying on the predefined harmful-token set: cross-modal harmful semantics becomes distinguishable only after the safety-critical window.

2. Refusal control: early text-only separability persists without refusal (Appendix G.3, Fig. 23).

We freeze the text-only probes trained on refused harmful text and benign text, and evaluate them on independent successfully jailbroken harmful text and normally answered benign text. Neither test group is refused. Pre-window AUROC nevertheless remains 0.963, 0.934, and 0.983, closely matching the original text-only setting (0.962, 0.934, 0.982). These results indicate that early text-only harmful–benign separability is not solely attributable to refusal behavior.

Table R5. Harmful–benign probe AUROC immediately before the safety-critical window (95% confidence intervals in brackets).

ModelLayerText-onlyCross-modalText-only: non-refusal test
LLaVA-1.5-7B120.962 [0.947, 0.976]0.600 [0.546, 0.661]0.963 [0.955, 0.971]
Qwen2.5-VL-7B110.934 [0.913, 0.953]0.569 [0.523, 0.622]0.934 [0.923, 0.945]
InternVL2-8B130.982 [0.973, 0.990]0.581 [0.541, 0.627]0.983 [0.977, 0.987]

Ref:

[1] Understanding and Rectifying Safety Perception Distortion in VLMs. NeurIPS 2025.

[2] OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities. EMNLP 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions and Responses to Major Concerns (Part 2/2)

II. Responses to the Reviewers' Major Concerns

1. Strengthened validation of CSD (Sec. 3.1 and Appendices G.1, G.3, and G.6).

We strengthen CSD validation in three ways: cross-model generalization, independent validation of Harmfulness Emergence, and analysis of the relationship between CSD and jailbreak outcomes. First, we extend CSD analysis and CURE evaluation to LLaVA-1.5-13B and Qwen3-VL-8B, observing CSD in both models while maintaining CURE's safety gains and general multimodal capabilities.

Second, we introduce layer-wise linear probing on hidden states, independent of the predefined harmful-token set and LM head. The probes confirm delayed cross-modal harmful–benign separability, which is not simply attributable to refusal behavior, as further verified using successfully jailbroken harmful texts and normally answered benign texts. Finally, comparisons between successfully refused and successfully jailbroken queries reveal stronger Safety Activation and earlier Harmfulness Emergence in the refused group. Together, these findings strengthen the evidence for CSD's cross-model generalizability, independent validity, and association with jailbreak outcomes.

2. Strengthened evaluation design, reliability, and cross-attack generalization (Appendix A.1, Sec. 5, and Appendices B and G.7).

We clarify data separation across identification, calibration, hyperparameter selection, and evaluation, specify the automatic evaluator and scoring protocols, and verify evaluator reliability through human annotations. We further evaluate CURE against two additional typographic attacks, MML-WR and MML-M, achieving ASR below 3% across all three LVLMs. These additions strengthen evaluation reliability and provide evidence of CURE's generalization to unseen attack constructions.

3. Expanded helpfulness and over-refusal analysis (Secs. 5.1–5.2 and 7, and Appendices G.4–G.5).

We expand the helpfulness and over-refusal analysis from two aspects. First, we add evaluations on MOSSBench and MM-SafetyBench++ Safe to assess CURE on benign but safety-sensitive queries. Results show modest increases in MOSSBench refusal rates (1.24–2.67 percentage points), while MM-SafetyBench++ Safe maintains high answer rates (97.18–98.31%) with negligible changes in response quality. These results indicate that CURE largely preserves helpfulness on benign but safety-sensitive queries, with a limited over-refusal trade-off.

Second, we conduct a blinded human audit of MSSBench Safe, distinguishing helpful responses incorrectly scored by the automatic evaluator from genuine false refusals, whose rates increase by 4.40–18.00 percentage points across the three models. Further diagnostics identify unsupported safety judgments as the dominant source of LLaVA's over-refusals (80.82%). These findings clarify the sources of the observed helpfulness degradation and highlight limitations in situational safety understanding as a key contributor to unnecessary refusals.

4. Extended inference-efficiency evaluation (Sec. 5.5 and Appendix G.2).

We extend latency and FLOPs evaluations to Qwen2.5-VL-7B and InternVL2-8B. CURE introduces only 1.55% and 1.42% additional latency, respectively, with modest FLOPs overhead (4.60% and 9.87%). Among the evaluated defenses, CURE also requires the fewest FLOPs on both models. These results demonstrate CURE's computational efficiency across different LVLM architectures.

5. Improved visualization and presentation (Fig. 1 in the main text, Appendix F.3, and throughout the manuscript).

We add visualizations of SCARE-selected visual tokens at layers 4, 8, and 12, showing that the selected patches consistently correspond to safety-relevant image regions across examples from MM-SafetyBench and SIUO. These results provide qualitative evidence that SCARE identifies visual cues relevant to contextual safety assessment. We also refine figure layouts and standardize component terminology throughout the manuscript.

Overall, these revisions provide additional experimental evidence and more detailed analyses addressing the reviewers' concerns. We provide the corresponding results and further explanations in the revised manuscript and our point-by-point responses.

AuthorsOpenReview ↗

General Response: Summary of Revisions and Responses to Major Concerns (Part 1/2)

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. In response, we have substantially revised and expanded the manuscript through additional experiments, in-depth analyses, and improvements to the presentation. The revision substantially expands the supplementary material, incorporating approximately 8 additional pages, alongside 8 new revised figures (17 panels in total) and 8 new tables, while keeping the main paper within the 10-page limit. The following sections summarize the revisions incorporated into the manuscript, the additional evidence provided in our responses, and how these collectively address the reviewers' major concerns.

I. Overview of Manuscript Revisions

Changes incorporated into the revised manuscript (marked in red in the revised PDF)

  1. Sec. 3.1 and Appendix G.1: CSD analysis is extended to two additional LVLMs, LLaVA-1.5-13B and Qwen3-VL-8B, with additional safety and utility evaluations.
  2. Sec. 5.5 and Appendix G.2: Inference-efficiency evaluations are extended to Qwen2.5-VL-7B and InternVL2-8B, including latency and FLOPs comparisons.
  3. Sec. 3.1 and Appendix G.3: Vocabulary-independent hidden-state probing and refusal-controlled evaluation are introduced to independently examine delayed cross-modal harmful-semantic emergence.
  4. Sec. 5.2 and Appendix G.4: Helpfulness evaluations on MOSSBench and MM-SafetyBench++ Safe are added, reporting refusal rates, answer rates, and response quality.
  5. Secs. 5.1 and 7, and Appendix G.5: A blinded human audit of MSSBench Safe, false-refusal categorization, and situational safety-judgment diagnostics are added.
  6. Sec. 3.1 and Appendix G.6: Successfully refused and jailbroken cross-modal queries are compared to further examine the relationship between CSD and jailbreak outcomes.
  7. Appendices A.1 and G.7, and Sec. 5: Data usage and evaluation splits are clarified, and generalization to two additional typographic attacks, MML-WR and MML-M, is evaluated across three LVLMs.
  8. Fig. 1, Appendix F.3, and throughout the manuscript: SCARE-selected visual tokens are visualized at different layers, figure layouts are refined, and component terminology is standardized.

Additional evidence provided in the reviewer responses: Human verification of 300 responses (91.00% agreement with GPT-4o-mini) and a controlled comparison against generic refusal boosting are provided in Tables R15 and R17, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.