- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates safety failures in large vision-language models (LVLMs) under cross-modal jailbreak attacks. It proposes the "Cross-modal Safety Desynchronization" (CSD) hypothesis, which suggests that harmful semantics emerge only in deeper layers, thereby bypassing the critical safety-enforcement window located in the intermediate layers. Based on this insight, the authors introduce CURE, a method that improves model safety responses by selectively enhancing early-stage cross-modal attention and conditionally boosting the activation of safety-related MLP neurons.
Critical strengths
- Research questions have value. Linking the inter-layer relationship between cross-modal information fusion and rejection calculation provides a perspective worth exploring for understanding multi-modal safety failures.
- Experiments show real benefits. CURE improves results across multiple models and safety benchmarks, and component ablation also demonstrates that attentional and neuronal interventions have complementary effects.
- The ablation coverage is relatively sufficient, including intervention layer range, visual token selection, activation enhancement strategy and hyperparameter sensitivity, which is helpful for understanding the design choices of the method.
Critical weaknesses
- Equation (4) in Section 3.1 maps intermediate-layer representations to the vocabulary via the LM head and then calculates their similarity to pre-defined harmful-word vectors. This metric measures lexical tendencies under a specific readout mechanism; it cannot be directly equated to the actual formation of harmful semantics. A lower score might indicate that risky information cannot yet be decoded by the final LM head, or it could simply mean the model employs a different internal representation—not necessarily that it lacks an understanding of the risk. Furthermore, this metric may be influenced by the specific phrasing of the model's responses. The token heatmaps in the appendix rely on this same measurement approach and do not offer independent verification. Consequently, while the current results support the claim that "signals related to harmful words emerge late," they are insufficient to confirm that "an understanding of harm itself emerges late."
- In Table 1, CURE shows a significant decline in accuracy on the safe samples from the MSSBench dataset: from 92.50% to 54.67% for LLaVA, from 96.67% to 81.67% for Qwen, and from 88.16% to 72.00% for InternVL2. These results indicate that gains on unsafe samples come at the cost of the model's ability to respond appropriately to safe inputs. While accuracy alone cannot determine whether these losses are entirely due to false refusals, the authors should provide an analysis of the types of errors occurring. Scores on general-purpose vision-language benchmarks remain largely unchanged, but these cannot serve as a substitute for evaluating performance on requests that are sensitive from a safety perspective yet benign in reality. It is recommended that the authors report false-refusal rates and compare different methods while controlling for accuracy—or the capacity for appropriate responses—on safe samples.
Questions for authors
see Weaknesses
5 comments
Response to Reviewer gvsB (Part 2/3)
W2. MSSBench Safe errors and safety–helpfulness trade-offs
"CURE shows a significant decline in accuracy on the safe samples from the MSSBench dataset [...] The authors should provide an analysis of the types of errors occurring and report false-refusal rates."
Thanks for this suggestion. The decline in MSSBench Safe accuracy reflects both a scoring artifact and genuine false refusals, which are largest on LLaVA (24.33% vs. 6.33% for Baseline) and mostly stem from unsupported safety judgments. We answer the two requests in W2 across two parts:
- Part 2/3 (this part): false-refusal rates. In a human audit of MSSBench Safe, genuine false refusals rise by 18.00, 7.33, and 4.40 points on LLaVA, Qwen, and InternVL, versus official accuracy drops of 37.83, 15.00, and 16.16 points (Table R19). On dedicated benchmarks, MOSSBench refusal rates rise by only 1.24-2.67 points (Table R20).
- Part 3/3: error types. On LLaVA, 80.82% of false refusals are unsupported safety judgments, and we analyze their cause and the safety–helpfulness trade-off.
1. MSSBench Safe: scoring artifact versus genuine false refusals (Table R19).
The official drop has two causes:
- Cause 1: Scoring artifact. MSSBench's automatic judge marks a response incorrect whenever it flags a safety risk, even if it fully answers the request. For example, given a photo of a burning fire pit and the question "What materials are best for starting a campfire?", a response lists dry wood and kindling and adds that the fire should be supervised, yet is scored incorrect (Appendix G.5, Fig. 25).
- Cause 2: Genuine false refusals. CURE refuses some safe requests, mainly on LLaVA, as analyzed in Part 3/3.
To separate them, we conducted a human verification, following [1], which identified the same scoring problem. Three annotators labeled each response as an appropriate answer, an unnecessary refusal, or another failure, given the image and question and blind to method and original score (majority vote). Genuine false refusals rise by 18.00, 7.33, and 4.40 points, about half or less of the official drops. The scoring artifact accounts for the remaining 19.83, 7.67, and 11.76 points.
Table R19. MSSBench Safe scoring and human audit (%).
| Model | Method | Original Safe Acc. ↑ | Human appropriate responses ↑ | False refusals ↓ | Helpful answers scored incorrect |
|---|---|---|---|---|---|
| LLaVA-1.5-7B | Baseline | 92.50 | 93.67 | 6.33 | 1.17 |
| LLaVA-1.5-7B | CURE | 54.67 | 75.67 | 24.33 | 21.00 |
| Qwen2.5-VL-7B | Baseline | 96.67 | 96.83 | 3.17 | 0.16 |
| Qwen2.5-VL-7B | CURE | 81.67 | 89.50 | 10.50 | 7.83 |
| InternVL2-8B | Baseline | 88.16 | 95.20 | 4.80 | 7.04 |
| InternVL2-8B | CURE | 72.00 | 90.80 | 9.20 | 18.80 |
2. False refusals on dedicated benign-but-sensitive benchmarks (Table R20).
We also evaluate Baseline and CURE on MOSSBench (refusal rate) and MM-SafetyBench++ Safe (answer rate and response quality). CURE raises MOSSBench refusal rates by only 2.00, 2.67, and 1.24 points on LLaVA, Qwen, and InternVL, while MM-SafetyBench++ Safe answer rates stay at 97.18-98.31% with quality changes of -0.07 to +0.03. These benchmarks show largely preserved helpfulness, so the larger cost on LLaVA is specific to MSSBench Safe. We have revised Section 5 to separate general capability retention from helpfulness on safety-sensitive benign requests, and to report MSSBench Safe and Unsafe separately.
Table R20. False refusals and helpfulness on benign but safety-sensitive requests.
| Model | Method | MOSSBench refusal (%) ↓ | MM-SafetyBench++ Safe answer (%) ↑ | Quality ↑ |
|---|---|---|---|---|
| LLaVA-1.5-7B | Baseline | 19.00 | 97.96 | 3.63 |
| LLaVA-1.5-7B | CURE | 21.00 | 98.31 | 3.66 |
| Qwen2.5-VL-7B | Baseline | 3.00 | 98.80 | 4.36 |
| Qwen2.5-VL-7B | CURE | 5.67 | 97.68 | 4.29 |
| InternVL2-8B | Baseline | 9.43 | 96.98 | 4.25 |
| InternVL2-8B | CURE | 10.67 | 97.18 | 4.23 |
Ref:
[1] Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model. EMNLP 2025.
→ Next (Part 3/3): error types and causes of LLaVA's false refusals, and the safety–false-refusal trade-off.
Response to Reviewer gvsB (Part 3/3)
W2 (continued). Error types, causes, and the safety–false-refusal trade-off
3. Error types of LLaVA's false refusals (Tables R21-R22; Appendix G.5).
We classify the 146 false refusals from Table R19 by their stated grounds (Table R21). 80.82% are unsupported safety judgments, where a safe scene is called unsafe without a valid reason. The rest are incorrect visual facts (10.96%) and refusals without a specific reason (8.22%). Since CURE builds on the model's own multimodal understanding, we hypothesize that LLaVA's misjudgments of safe scenes become refusals under safety enhancement. Two diagnostics support this (Table R22):
- LLaVA already misjudges these scenes. We ask the original LLaVA whether the requested activity is safe in the pictured scene [1]. On 118 safe queries that CURE refuses with unsupported safety judgments, it answers "unsafe" for 85.59%, versus 18.64% on 118 safe queries that CURE answers normally.
- Correcting the scene understanding reduces refusals. Adding verified factual scene descriptions, without changing images or tasks, lowers CURE's refusal rate on the failure subset from 100.00% to 33.90% [1, 2].
These results are associative: they suggest that LLaVA's limited situational safety judgment contributes to CURE's false refusals.
Table R21. LLaVA's audited false-refusal types.
| Error type | Count | Proportion (%) |
|---|---|---|
| Incorrect visual facts | 16 | 10.96 |
| Unsupported safety judgments | 118 | 80.82 |
| No specific explanation | 12 | 8.22 |
| Total | 146 | 100.00 |
Table R22. LLaVA judgment and scene-description diagnostic.
| Safe query group | N | Baseline judgment error (%) | CURE refusal before (%) | CURE refusal after (%) |
|---|---|---|---|---|
| Unsupported-judgment refusals | 118 | 85.59 | 100.00 | 33.90 |
| Normally answered controls | 118 | 18.64 | 0.00 | 0.00 |
Before/after: without/with factual descriptions. Baseline judgment is tested separately on original inputs.
4. Safety–helpfulness trade-off (Table R23).
On LLaVA, we compare CURE with a fixed neuron boost that uses the same neurons and window but no SCARE. A strong boost (α=10) reaches 2.24% ASR but raises MOSSBench refusal to 60.33%, whereas CURE reaches 1.78% ASR with 21.00% refusal. This is not an exact refusal-matched comparison.
Table R23. LLaVA safety–false-refusal trade-off (%).
| Method | MOSSBench refusal ↓ | MM-SafetyBench ASR ↓ |
|---|---|---|
| Unboosted (α=0) | 19.00 | 53.85 |
| Fixed boost (α=10) | 60.33 | 2.24 |
| CURE | 21.00 | 1.78 |
Ref:
[1] Multimodal Situational Safety. ICLR 2025.
[2] MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries? ICLR 2025.
Response to Reviewer gvsB (Part 1/3)
We thank reviewer gvs8 for the constructive feedback and address each concern below.
Changes in the revised paper that relate to this review (marked in
redin the uploaded PDF)W1. Independent validation of harmfulness timing
Thanks for highlighting this concern. To test whether the observed delay persists beyond this readout, we add layer-wise linear probing on hidden states, which uses neither the predefined harmful-token set nor the LM head (Appendix G.3, Fig. 22). Linear probing is a widely used and well-established method for testing whether a model’s internal representations encode a concept [1], and has been applied to study harmfulness and refusal representations in LLMs [2, 3]. Across all three models, text-only harmful–benign separability is already high before the safety-critical window, whereas cross-modal separability rises toward its end and beyond, consistent with CSD.
Readout-independent validation (Appendix G.3, Fig. 22). We train layer-wise linear probes on the hidden states at the final input token to distinguish harmful inputs from
D_pairand benign inputs fromD_benign, and evaluate on held-out samples. For each model and layer, we train separate probes on text-only and cross-modal data and evaluate each on held-out data of the same modality. Because a probe decodes harmful–benign information from the representation itself, it does not require that information to be expressed as harmful words under the LM head. Moreover, these hidden states are computed before any response is generated, so the measurement does not depend on the phrasing of the generated response. Table R18 reports AUROC immediately before the safety-critical window.** Text-only AUROC reaches 0.962, 0.934, and 0.982, compared with 0.600, 0.569, and 0.581 for cross-modal inputs on LLaVA, Qwen, and InternVL, respectively. The full layer-wise curves show cross-modal separability increasing toward the end of and beyond the window (Appendix G.3, Fig. 22). CSD therefore persists under a readout that is independent of the LM head, the word set, and response phrasing: cross-modal harmful semantics becomes distinguishable only after the safety-critical window.Table R18. Harmful–benign probe AUROC immediately before the safety-critical window.
Ref:
[1] Understanding and Rectifying Safety Perception Distortion in VLMs. NeurIPS 2025.
[2] Improving Alignment and Robustness with Circuit Breakers. NeurIPS 2024.
[3] LLMs Encode Harmfulness and Refusal Separately. NeurIPS 2025.